Voice query refinement to embed context in a voice query
Systems and methods are described for providing contextual search results. The system may receive a search query during presentation of a video. If the query is ambiguous, the system accesses some of the frames of the video. The frames are analyzed to identify a performed action depicted in the frames. The system retrieves a keyword related to the identified action. The ambiguous query is augmented with the keyword. The augmented search query is used to search for and output relevant search results.
1. A method for providing contextual search results to ambiguous queries, the method comprising:
receiving a search query during a presentation of a video;
determining whether at least one word in the search query is ambiguous;
in response to determining that at least one word in the search query is ambiguous:
(i) accessing a plurality of frames from the video that were presented concurrently with receiving the search query;
(ii) analyzing the plurality of frames to identify a performed action;
(iii) retrieving a keyword associated with the identified action; and
performing a text-based search based on the search query and the keyword; and
outputting results of the search.
2. The method of claim 1 ,
wherein identifying the performed action comprises:
identifying a character in each of the plurality of frames;
generating a model of the identified character's movements;
determining that the generated model matches a movement template; and
wherein retrieving the keyword associated with the identified action comprises retrieving metadata of the movement template.
3. The method of claim 2 , wherein generating a model of the identified character's movements comprises:
identifying body parts of the identified character; and
calculating an angle between two body parts of the identified character.
4. The method of claim 3 , wherein determining that the generated model matches the movement template comprises:
comparing the calculated angle with a reference angle of the movement template; and
in response to determining that the calculated angle matches a reference angle determining that the generated model matches the movement template.
5. The method of claim 1 , wherein receiving the search query comprises:
detecting user voice input; and
performing speech to text analysis of the user voice input to derive the search query.
6. The method of claim 1 , wherein determining that at least one word in the search query is ambiguous comprises:
determining that the search query comprises at least one of: a pronoun and an auxiliary verb.
7. The method of claim 1 , wherein accessing the plurality of frames from the video that were presented concurrently with receiving the search query comprises:
receiving an audio sample of the video that were presented concurrently with receiving the search query;
identifying a time location in the video where the audio sample occurred; and
extracting frames corresponding to the time location.
8. The method of claim 7 , wherein extracting frames corresponding to the time location comprises:
extracting frames from a predetermined time period prior to the time location; and
extracting frames from a predetermined time period after the time location.
9. The method of claim 1 , wherein accessing the plurality of frames from the video that were presented concurrently with receiving the search query comprises capturing displayed frames of the video for a predetermined time period after receiving the search query.
10. The method of claim 1 , wherein accessing the plurality of frames from the video that were presented concurrently with receiving the search query comprises capturing a predetermined number of displayed frames after receiving the search query.
11. A system for providing contextual search results to ambiguous queries, the system comprising;
input circuitry of a device configured to:
receive a search query during a presentation of a video; and
control circuitry of the device configured to:
determine whether at least one word in the search query is ambiguous;
in response to determining that at least one word in the search query is ambiguous:
(i) access a plurality of frames from the video that were presented concurrently with receiving the search query;
(ii) analyze the plurality of frames to identify a performed action;
(iii) retrieve a keyword associated with the identified action; and
perform a text-based search based on the augmented search query and the keyword; and
output results of the search.
12. The system of claim 11 ,
wherein, when identifying the performed action, the control circuitry is configured to:
identify a character in each of the plurality of frames;
generate a model of the identified character's movements;
determine that the generated model matches a movement template; and
wherein, when retrieving the keyword associated with the identified action, the control circuitry is configured to retrieve metadata of the movement template.
13. The system of claim 12 , wherein, when generating a model of the identified character's movements, the control circuitry is configured to:
identify body parts of the identified character; and
calculate an angle between two body parts of the identified character.
14. The system of claim 13 , wherein, when determining that the generated model matches the movement template, the control circuitry is configured to:
compare the calculated angle with a reference angle of the movement template; and
in response to determining that the calculated angle matches a reference angle, determine that the generated model matches the movement template.
15. The system of claim 11 , wherein, when receiving the search query, the control circuitry is configured to:
detect user voice input; and
perform speech to text analysis of the user voice input to derive the search query.
16. The system of claim 11 , wherein, when determining that at least one word in the search query is ambiguous, the control circuitry is configured to:
determine that the search query comprises at least one of: a pronoun and an auxiliary verb.
17. The system of claim 11 , wherein, when accessing the plurality of frames from the video that were presented concurrently with receiving the search query, the control circuitry is configured to:
receive an audio sample of the video that were presented concurrently with receiving the search query;
identify a time location in the video where the audio sample occurred; and
extract frames corresponding to the time location.
18. The system of claim 17 , wherein, when extracting frames corresponding to the time location, the control circuitry is configured to:
extract frames from a predetermined time period prior to the time location; and
extract frames from a predetermined time period after the time location.
19. The system of claim 11 , wherein, when accessing the plurality of frames from the video that were presented concurrently with receiving the search query, the control circuitry is configured to capture displayed frames of the video for a predetermined time period after receiving the search query.
20. The system of claim 11 , wherein, when accessing the plurality of frames from the video that were presented concurrently with receiving the search query, the control circuitry is configured to capture a predetermined number of displayed frames after receiving the search query.