Combining Modalities for Hyperlinking Navigation in Multimedia Collections
Date:
Hyperlinks connect related content, as in the World Wide Web which uses links to help people find referenced webpages. In this talk I examine using hyperlinks between related video segments in order to enable users to browse video collections. I will focus on how hyperlinks between video segments can be automatically constructed using information retrieval methods. Working with video recordings enables combining different modalities to improve link quality, and thus to improve information access. Using context and metadata together with spoken content automatically mined from video using automatic speech recognition can achieve a relative improvement of 180% over spoken content alone. A further relative improvement of almost 9% can be achieved by additionally leveraging visual similarity. These approaches were tuned and tested on benchmark tasks organized at MediaEval and TRECVID, and they ranked first in the MediaEval Search and Hyperlinking task.