Granted Patent B2

US 9,953,649 · App. 15/430,952 · Granted Apr 24, 2018

Abstract

Claims (73)

1. A method for processing one or more multi-modal device interactions in a natural language voice services environment that includes a plurality of electronic devices each separate from one another, the plurality of electronic devices including a first electronic device having at least a non-voice input device and a second electronic device having at least a voice input device, the method comprising:

receiving, by a voice-click component of at least one of the plurality of electronic devices, a non-voice interaction detected at the first electronic device and a natural language utterance detected at the second electronic device;

determining, by the voice-click component, first context information relating to the non-voice interaction, wherein the first context information includes context relating to the non-voice interaction;

determining, by the voice-click component, second context information relating to the natural language utterance, wherein the second context information includes context relating to the natural language utterance;

determining, by the voice-click component, an intent based on the first context relating to the non-voice interaction and the second context relating to the natural language utterance;

generating, by the voice-click component, a request based on the determined intent; and

transmitting, by the voice-click component, the request to a target electronic device from among the plurality of electronic devices.

2. The method of claim 1 , the method further comprising:

generating, by the first electronic device, a user interface, wherein receiving the non-voice interaction comprises:

receiving, by the first electronic device, a point of focus on the user interface, and wherein the first context is based on the point of focus.

3. The method of claim 2 , wherein the user interface comprises a map display interface and wherein the point of focus includes a location on the map display that indicates a geolocation, which is used as the first context.

4. The method of claim 3 , the method further comprising:

generating, by the voice-click component, a transaction lead based on the geolocation; and

displaying, on the first electronic device, the transaction lead as a selectable display option on the map display.

5. The method of claim 1 , the method further comprising:

storing, by the voice-click component, a plurality of types of nonvoice interactions that are recognized by the voice-click component;

receiving, by the voice-click component, a new type of nonvoice interaction to be recognized; and

storing, by the voice-click component, the new type of nonvoice interaction with the plurality of types of nonvoice interactions.

6. The method of claim 1 , the method further comprising:

storing, by the voice-click component, a list of the plurality of electronic devices from which the voice-click components is configured to receive nonvoice interactions and/or voice interactions;

monitoring, by the voice-click component, the plurality of electronic devices based on the list.

7. The method of claim 6 , the method further comprising:

determining, by the first electronic device, a first time at which the nonvoice interaction was detected at the first electronic device;

determining, by the second electronic device, a second time at which the natural language utterance was detected at the first electronic device;

obtaining, by the voice-click component, the first time and the second time; and

determining, by the voice-click component, that the nonvoice interaction and the natural language utterance relate form a multi-modal interaction that is to be interpreted together based on the first time and the second time.

8. The method of claim 6 , the method further comprising:

receiving, by the voice-click component, and indication of a new electronic device to be added to the list of the plurality of electronic devices;

adding, by the voice-click component, the new electronic device to the list of the plurality of electronic device; and

establishing, by the voice-click component, a new listener to monitor the new electronic device.

9. The method of claim 1 , the method further comprising:

obtaining, by the voice-click component, a constellation model that provides knowledge relating to content, services, applications, and/or intent determination capabilities of each of the plurality of electronic devices; and

identifying, by the voice-click component, the target electronic device based on the constellation model and the request based on a determination that the target electronic device is able to handle the request as specified in the constellation model, wherein the request is transmitted to the target electronic device.

10. The method of claim 1 , the method further comprising:

obtaining, by the voice-click component, one or more words or phrases of the natural language utterance; and

determining, by the voice-click component, the intent based on the one or more words or phrases, the first context, and the second context.

11. The method of claim 1 , wherein the voice-click component executes on the first electronic device, the second electronic device, or another one of the plurality of electronic devices.

12. A system for processing one or more multi-modal device interactions in a natural language voice services environment that includes a plurality of electronic devices each separate from one another, the plurality of electronic devices including a first electronic device having at least a non-voice input device and a second electronic device having at least a voice input device, the system:

a computer system comprising one or more physical processors implementing a voice-click component to:

receive a non-voice interaction detected at the first electronic device and a natural language utterance detected at the second electronic device;

determine first context information relating to the non-voice interaction, wherein the first context information includes context relating to the non-voice interaction;

determine second context information relating to the natural language utterance, wherein the second context information includes context relating to the natural language utterance;

determine an intent based on the first context relating to the non-voice interaction and the second context relating to the natural language utterance;

generate a request based on the determined intent; and

transmit the request to a target electronic device from among the plurality of electronic devices.

13. The system of claim 12 , wherein the first electronic device is programmed to:

generate a user interface, wherein receiving the non-voice interaction comprises:

receive a point of focus on the user interface, and wherein the first context is based on the point of focus.

14. The system of claim 13 , wherein the user interface comprises a map display interface and wherein the point of focus includes a location on the map display that indicates a geolocation, which is used as the first context.

15. The system of claim 14 , wherein the voice-click component is further configured to:

generate a transaction lead based on the geolocation; and wherein the first electronic device is configured to:

display the transaction lead as a selectable display option on the map display.

16. The system of claim 12 , wherein the voice-click component is further configured to:

store a plurality of types of nonvoice interactions that are recognized by the voice-click component;

receive a new type of nonvoice interaction to be recognized; and

store the new type of nonvoice interaction with the plurality of types of nonvoice interactions.

17. The system of claim 12 , wherein the voice-click component is further configured to:

store a list of the plurality of electronic devices from which the voice-click components is configured to receive nonvoice interactions and/or voice interactions;

monitor the plurality of electronic devices based on the list.

18. The system of claim 17 , wherein the first electronic device is further configured to:

determine a first time at which the nonvoice interaction was detected at the first electronic device;

determine a second time at which the natural language utterance was detected at the first electronic device;

obtain the first time and the second time; and

determine that the nonvoice interaction and the natural language utterance relate form a multi-modal interaction that is to be interpreted together based on the first time and the second time.

19. The system of claim 17 , wherein the voice-click component is further configured to:

receive an indication of a new electronic device to be added to the list of the plurality of electronic devices;

add the new electronic device to the list of the plurality of electronic device; and

establish a new listener to monitor the new electronic device.

20. The system of claim 12 , wherein the voice-click component is further configured to:

obtain a constellation model that provides knowledge relating to content, services, applications, and/or intent determination capabilities of each of the plurality of electronic devices; and

identify the target electronic device based on the constellation model and the request based on a determination that the target electronic device is able to handle the request as specified in the constellation model, wherein the request is transmitted to the target electronic device.

21. The system of claim 12 , wherein the voice-click component is further configured to: obtain one or more words or phrases of the natural language utterance; and

determine the intent based on the one or more words or phrases, the first context, and the second context.

Assignments (10)