System and method for processing multi-modal device interactions in a natural language voice services environment
View Patent ↗A system and method for processing multi-modal device interactions in a natural language voice services environment may be provided. In particular, one or more multi-modal device interactions may be received in a natural language voice services environment that includes one or more electronic devices. The multi-modal device interactions may include a non-voice interaction with at least one of the electronic devices or an application associated therewith, and may further include a natural language utterance relating to the non-voice interaction. Context relating to the non-voice interaction and the natural language utterance may be extracted and combined to determine an intent of the multi-modal device interaction, and a request may then be routed to one or more of the electronic devices based on the determined intent of the multi-modal device interaction.
1. A computer-implemented method for processing a natural language utterance via multiple input modes, the method being implemented by a computer system that includes one or more processors executing one or more computer program instructions which, when executed, perform the method, the method comprising:
receiving, by the one or more physical processors, a first user input via a voice input mode and a second user input via a non-voice input mode, wherein the first user input includes a natural language utterance, and the second user input includes a non-voice input relating to the natural language utterance;
processing, by the one or more physical processors, the natural language utterance to recognize one or more words of the natural language utterance, wherein the one or more recognized words include a reference word;
identifying, by the one or more physical processors based on the one or more recognized words, a query;
identifying, by the one or more physical processors based on the non-voice input, context information for the one or more recognized words, wherein the context information indicates context for the reference word;
determining, by the one or more physical processors based on the context information, one or more interpretations of the one or more recognized words, wherein determining the one or more interpretations comprises:
identifying the reference word;
identifying, based on the reference word context, a product or service to which the reference word refers; and
determining an meaning of the reference word based on the identification of the product or service; and
generating, by the one or more physical processors based on the one or more interpretations, a response to the query.
2. The method of claim 1 , wherein the context information is identified further based on the natural language utterance.
3. The method of claim 1 , further comprising:
determining, by the one or more physical processors, prior context information associated with one or more prior natural language utterances, wherein the one or more prior natural language utterances are received by the one or more physical processors before the natural language utterance is received, and
wherein generating the response comprises generating the response based on the one or more interpretations and the prior context information.
4. The method of claim 1 , wherein the natural language utterance identifies a domain of the query, and wherein the response is generated based on the domain.
5. The method of claim 1 , wherein the non-voice input identifies data, a segment, item, or a point of focus on the another input device, and wherein the data, segment, item, or point of focus provides the context information.
6. The method of claim 1 , wherein the non-voice input identifies a point of focus on a touch screen, and wherein the identifying the context information comprises identifying a location based on the point of focus on the touch screen.
7. The method of claim 6 , wherein the query relates to the product or service, and wherein the generating the response comprises retrieving information on the product or service with respect to the location.
8. The method of claim 1 , wherein the generating the response comprises routing the query to a device or application configured to process the query.
9. The method of claim 1 , wherein the identifying the query comprises:
identifying one or more interrogative pronouns in the natural language utterance; and
identifying the query based on the one or more interrogative pronouns.
10. The method of claim 1 , further comprising:
receiving, by the one or more physical processors, a third user input;
identifying, by the one or more physical processors, second context information from the third user input, wherein the context information is different than the second context information; and
generating, by the one or more physical processors, a response to the third user input based on the identified query and based on the second context information.
11. The method of claim 10 , wherein the first user input identifies a domain of the query, wherein the second user input identifies a location associated with the query, and wherein the third user input identifies a different location associated with the query.
12. A device for processing a natural language utterance via multiple input modes, comprising:
one or more physical processors programmed to execute one or more computer program instructions which, when executed, cause the one or more physical processors to:
receive a first user input via a voice input mode and a second user input via a non-voice input mode, wherein the first user input includes a natural language utterance, and the second user input includes a non-voice input relating to the natural language utterance;
process the natural language utterance to recognize one or more words of the natural language utterance, wherein the one or more recognized words include a reference word;
identify, based on the one or more recognized words, a query;
identify, based on the non-voice input, context information for the one or more recognized words, wherein the context information indicates context for the reference word;
determine, based on the context information, one or more interpretations of the one or more words, wherein determining the one or more interpretations comprises:
identifying the reference word;
identifying, based on the reference word context, a product or service to which the reference word refers; and
determining an meaning of the reference word based on the identification of the product or service; and
generate, based on the one or more interpretations, a response to the query.
13. A non-transitory computer-readable medium for processing a natural language utterance via multiple input modes, the non-transitory computer-readable medium comprising one or more instructions that, when executed by one or more processors, cause the one or more processors to:
receive a first user input via a voice input mode and a second user input via a non-voice input mode, wherein the first user input includes a natural language utterance, and the second user input includes a non-voice input relating to the natural language utterance;
process the natural language utterance to recognize one or more words of the natural language utterance, wherein the one or more recognized words include a reference word;
identify, based on the one or more recognized words, a query;
identify, based on the non-voice input, context information for the one or more recognized words, wherein the context information indicates context for the reference word;
determine, based on the context information, one or more interpretations of the one or more recognized words, wherein determining the one or more interpretations comprises:
identifying the reference word;
identifying, based on the reference word context, a product or service to which the reference word refers; and
determining an meaning of the reference word based on the identification of the product or service; and
generate, based on the one or more interpretations, a response to the query.
14. A computer-implemented method of processing a natural language utterance via multiple input modes, the method being implemented by a computer system that includes one or more physical processors executing one or more computer program instructions which, when executed, perform the method, the method comprising:
receiving, by the one or more physical processors, a first user input via a voice input mode and a second user input via a non-voice input mode, wherein the first user input includes a natural language utterance, and the second user input includes a non-voice input relating to the natural language utterance;
processing, by the one or more physical processors, the natural language utterance to recognize one or more words of the natural language utterance, wherein the one or more recognized words include a reference word;
identifying, by the one or more physical processors based on the one or more recognized words, a command;
identifying, by the one or more physical processors based on the non-voice input, context information for the one or more recognized words, wherein the context information indicates context for the reference word;
determining, by the one or more physical processors based on the context information, one or more interpretations of the one or more recognized words, wherein determining the one or more interpretations comprises:
identifying the reference word;
identifying, based on the reference word context, a product or service to which the reference word refers; and
determining an meaning of the reference word based on the identification of the product or service; and
generating, by the one or more physical processors based on the one or more interpretations, a command signal associated with the command.
15. The method of claim 14 , wherein the command includes a multimedia processing command.
16. The method of claim 15 , wherein the context information identifies multimedia associated with the multimedia processing command.