Hyper video: information retrieval using realtime buffers
View Patent ↗Disclosed is a method and device for selecting documents, such as Web pages or sites, for presentation to a user, in response to a user expression of interest, during the course of presentation to the user of a document, such as a video or audio selection, whose content varies with time. The method takes advantage of information retrieval techniques to select documents related to the portion of the temporal document in which the user has expressed interest. The method generates the search query to use to select documents by reference to text associated with the portion of the temporal document in which the user has expressed interest, as by using the closed caption test associated with the video, or by using speech recognition techniques. The method further uses a weighting function to weigh the terms used in the search query, depending on their temporal relationship to the user expression of interest. The method uses a buffer to save necessary information about the temporal occurrence of terms in the temporal document.
1. A computer implemented method for finding documents which relate to a portion of a temporal document, comprising:
(a) presenting a temporal document, wherein the temporal document is a time related document;
(b) determining text associated with portions of the temporal document, and
(c) storing in a buffer each term in said associated text, together with a time in the temporal document at which the term occurs;
(d) in response to a user signal of interest at a particular time during the temporal document, identifying a portion of the temporal document that occurred during a time interval before said user signal of interest;
(e) for a first term in said buffer occurring during said time interval, determining a first weight based on:
a function W(t) associated with said time interval, and
the time in the temporal document at which said first term occurs, wherein W(t) is equal for all times between t 1 before the signal of interest is given and t 2 before the signal is given, and decreases from t 1 until the time of the signal, and increases from a time t 3 before the signal is given to the time t 2 , and is zero for all other times;
(f) identifying related documents by use of information retrieval techniques as applied to said first term and said first weight.
2. The method of claim 1 , wherein the times at which the terms occur are determined using the synchronization information contained in the temporal document.
3. The method of claim 2 , wherein the temporal document uses the SMIL standard for synchronization of multimedia elements in a document.
4. The method of claim 1 , wherein the buffer has a fixed size such that it has sufficient capacity to hold all the terms which may occur in the temporal document during the time period that extends from when a signal of interest occurs to the earliest time prior to that for which the weighting function W(t) is nonzero.
5. The method of claim 4 , wherein the buffer has 8 kilobytes of capacity.
6. The method of claim 1 , wherein the size of the buffer is varied such that it has sufficient capacity at any given time to hold all terms which have occurred in the temporal document during a time period that extends back from the given time to an earliest time for which the weighting function W(t) would be nonzero for a signal of interest occurring at the given time.
7. The method of claim 1 , wherein terms and associated times are stored in the buffer as if the buffer were a continuous loop, such that the buffer is filled in order, and when full a next entry is inserted at the beginning of the buffer.
8. The method of claim 1 , wherein the temporal document is video or audio material.
9. The method of claim 8 , wherein the text stored in the buffer is determined by application of speech recognition techniques to the audio component of the portion of the temporal document identified.
10. The method of claim 8 , wherein the text stored in the buffer is the closed-captioned text associated with the portion of the temporal document identified.
11. The method of claim 1 , wherein the temporal document includes text.
12. The method of claim 11 , wherein the document text appearing to the user varies with time and the text selected is that portion of the temporal document identified.
13. The method of claim 12 , wherein the text includes news bulletins, weather, sports scores or stock transaction or pricing information.
14. The method of claim 1 , wherein the temporal document is being generated as it is being accessed.
15. The method of claim 1 , wherein W(t) is equal for all times between t 1 before the signal of interest is given and t 2 before the signal is given, and is zero for all other times.
16. The method of claim 15 , wherein t 1 is 2 seconds and t 2 is 30 seconds.
17. The method of claim 1 , wherein t 1 is 2 seconds, t 2 is 15 seconds, and t 3 is 30 seconds.
18. The method of claim 1 , wherein W(t) decreases linearly from t 1 until the time of the signal, and increases linearly from t 3 before the signal is given to t 2 .
19. The method of claim 18 , wherein t 1 is 2 seconds, t 2 is 15 seconds, and t 3 is 30 seconds.
20. A computer implemented device for finding documents which relate to a portion of a temporal document, comprising:
(a) means for presenting a temporal document, wherein the temporal document is a time related document;
(b) means for determining text associated with portions of the temporal document, and storing in a buffer each term in said associated text, together with a time in the temporal document at which the term occurs;
(c) means for identifying a portion of the temporal document, in response to a user signal of interest at a particular time during the temporal document, said portion of the temporal document occurring during a time interval before said user signal of interest;
(d) for a first term in said buffer occurring during said time interval, determining a first weight based on:
a function W(t) associated with said time interval, and
the time in the temporal document at which said first term occurs, wherein W(t) is equal for all time between t 1 before the signal of interest is given and t 2 before the signal is given, and decreases from t 1 until the time of the signal, and increases from a time t 3 before the signal is given to the time t 2 , and is zero for all other times;
(e) means for identifying related documents by use information retrieval techniques as applied to said first term and said first weight.
21. The device of claim 20 , wherein the times at which the terms occur are determined using the synchronization information contained in the temporal document.
22. The device of claim 21 , wherein the temporal document uses the SMIL standard for synchronization of multimedia elements in a document.
23. The device of claim 20 , wherein the buffer has a fixed size such that it has sufficient capacity to hold all the terms which may occur in the temporal document during the time period that extends from when a signal of interest occurs to the earliest time prior to that for which the weighting function W(t) is nonzero.
24. The device of claim 23 , wherein the buffer has 8 kilobytes of capacity.
25. The device of claim 20 , wherein the size of the buffer is varied such that it has sufficient capacity at any given time to hold all terms which have occurred in the temporal document during a time period that extends back from the given time to an earliest time for which the weighting function W(t) would be nonzero for a signal of interest occurring at the given time.
26. The device of claim 20 , wherein terms and associated times are stored in the buffer as if the buffer were a continuous loop, such that the buffer is filled in order, and when full a next entry is inserted at the beginning of the buffer.
27. The device of claim 20 , wherein the temporal document is video or audio material.
28. The device of claim 27 , wherein the text stored in the buffer is determined by application of speech recognition techniques to the audio component of the portion of the temporal document identified.
29. The device of claim 27 , wherein the text stored in the buffer is the closed-captioned text associated with the portion of the temporal document identified.
30. The device of claim 20 , wherein the temporal document includes text.
31. The device of claim 30 , wherein the document text appearing to the user varies with time and the text selected is that portion of the temporal document identified.
32. The device of claim 31 , wherein the text includes news bulletins, weather, sports scores or stock transaction or pricing information.
33. The device of claim 20 , wherein the temporal document is being generated as it is being accessed.
34. The device of claim 20 , wherein W(t) is equal for all times between t 1 before the signal of interest is given and t 2 before the signal is given, and is zero for all other times.
35. The device of claim 34 , wherein t 1 is 2 seconds and t 2 is 30 seconds.
36. The device of claim 20 , wherein t 1 is 2 seconds, t 2 is 15 seconds, and t 3 is 30 seconds.
37. The device of claim 20 , wherein W(t) decreases linearly from t 1 until the time of the signal, and increases linearly from t 3 before the signal is given to t 2 .
38. The device of claim 37 , wherein t 1 is 2 seconds, t 2 is 15 seconds, and t 3 is 30 seconds.