Unified inverted index for video passage retrieval
View Patent ↗A method for information retrieval includes extracting from a video document visual data items and textual data items that occur in the document at respective occurrence times. Indexing records, which index both the visual and the textual data items by their respective occurrence times, are constructed and stored in a memory.
1. A method for information retrieval, comprising:
extracting from a video document visual data items and textual data items that occur in the document at respective occurrence times;
constructing and storing in a memory indexing records, which index both the visual and the textual data items by their respective occurrence times;
accepting a search query comprising a visual query term and a textual query term;
based on the occurrence times stored in the indexing records, identifying in the video document at least one time interval that contains at least one visual data item matching the visual query term and at least one textual data item matching the textual query term, wherein identifying the at least one time interval comprises identifying one or more time intervals, each of which contains a respective first occurrence of the visual data item and a respective second occurrence of the textual data item and meets a predefined temporal proximity criterion between the first and second occurrences, as the at least one time interval; and
reporting the at least one identified time interval in response to the search query.
2. The method according to claim 1 , wherein extracting the textual data items comprises extracting at least one of the textual data items from transcribed speech of the video document.
3. The method according to claim 1 , wherein extracting the textual data items comprises extracting at least one of the textual data items from metadata of the video document.
4. The method according to claim 1 , wherein extracting the visual data items comprises representing a selected image from the video document using one or more textual tokens, and defining one or more of the visual data items as the textual tokens.
5. The method according to claim 1 , wherein identifying the at least one time interval comprises scanning the indexing records to identify the first and second occurrences and, during the scanning, retaining only the time intervals that meet the temporal proximity criterion and discarding the time intervals that do not meet the temporal proximity criterion.
6. The method according to claim 1 , wherein identifying the at least one time interval comprises assigning respective scores to the at least one identified time interval.
7. The method according to claim 6 , wherein assigning the scores comprises adjusting a score of a given time interval responsively to a distance between the respective first occurrence and the respective second occurrence contained in the given time interval.
8. A method for information retrieval, comprising:
extracting from a video document visual data items and textual data items that occur in the document at respective occurrence times;
constructing and storing in a memory indexing records, which index both the visual and the textual data items by their respective occurrence times, wherein constructing and storing the indexing records comprise indexing both the visual and the textual data items in a single inverted index;
accepting a search query comprising a visual query term and a textual query term;
based on the occurrence times stored in the indexing records, identifying in the video document at least one time interval that contains at least one visual data item matching the visual query term and at least one textual data item matching the textual query term; and
reporting the at least one identified time interval in response to the search query.
9. Apparatus for information retrieval, comprising:
an interface, which is operative to accept a video document that includes visual data items and textual data items, which occur in the video document at respective occurrence times; and
a processor, which is configured to extract the visual data items and the textual data items from the video document, to construct indexing records, which index both the visual and the textual data items by their respective occurrence times, to store the indexing records in a memory, to accept a search query comprising a visual query term and a textual query term, to identify in the video document, based on the occurrence times stored in the indexing records, at least one time interval, which contains at least one visual data item matching the visual query term and at least one textual data item matching the textual query term, and to report the at least one identified time interval in response to the search query, wherein the processor is configured to identify one or more time intervals, each of which contains a respective first occurrence of the visual data item and a respective second occurrence of the textual data item and meets a predefined temporal proximity criterion between the first and second occurrences, as the at least one time interval.
10. The apparatus according to claim 9 , wherein the processor is configured to index both the visual and the textual data items in a single inverted index.
11. The apparatus according to claim 9 , wherein the processor is configured to extract at least one of the textual data items from transcribed speech of the video document.
12. The apparatus according to claim 9 , wherein the processor is configured to extract at least one of the textual data items from metadata of the video document.
13. The apparatus according to claim 9 , wherein the processor is configured to represent a selected image from the video document using one or more textual tokens, and to define one or more of the visual data items as the textual tokens.
14. The apparatus according to claim 9 , wherein the processor is configured to scan the indexing records to identify the first and second occurrences and, during the scanning, to retain only the time intervals that meet the temporal proximity criterion and to discard the time intervals that do not meet the temporal proximity criterion.
15. The apparatus according to claim 9 , wherein the processor is configured to assign respective scores to the at least one identified time interval.
16. The apparatus according to claim 15 , wherein the processor is configured to adjust a score of a given time interval responsively to a distance between the respective first occurrence and the respective second occurrence contained in the given time interval.