Granted Patent B2

US 12,254,886 · App. 18/590,549 · Granted Mar 18, 2025

Inventors: Akshay Goel (Seattle, WA); Nitin Khandelwal (Sunnyvale, CA); Richard Park (Palo Alto, CA); Brian Chatham (Pleasanton, CA); Jonathan Eccles (San Francisco, CA); David Sanchez (Burlingame, CA); Dmytro Lapchuk (Mountain View, CA)

Assignee: GOOGLE LLC

G10L15/32G10L15/18G10L15/22G10L15/30

Abstract

Claims (66)

1. A system comprising:

at least one processor; and

memory storing instructions that, when executed, cause the at least one processor to be operable to:

receive audio data that captures a spoken utterance of a user, the audio data being generated by one or more microphones of a client device of the user, and the spoken utterance being directed to an automated assistant executed at least in part at the client device;

determine, based on processing the audio data, a plurality of first-party interpretations of the spoken utterance, each of the plurality of first-party interpretations being associated with a corresponding first-party predicted value indicative of a magnitude of confidence that each of the first-party interpretations are predicted to satisfy the spoken utterance;

identify a given third-party agent capable of satisfying the spoken utterance;

transmit, to the given third-party agent and over one or more networks, and based on processing the audio data, one or more structured requests that, when received, causes the given third-party to determine a plurality of third-party interpretations of the spoken utterance, each of the plurality of third-party interpretations being associated with a corresponding third-party predicted value indicative of a magnitude of confidence that each of the third-party interpretations are predicted to satisfy the spoken utterance;

receive, from the given third-party agent and over one or more of the networks, the plurality of third-party interpretations of the spoken utterance;

select, based on the corresponding first-party predicted values and the corresponding third-party predicted values, a given interpretation of the spoken utterance from among the plurality of first-party interpretations and the plurality third-party interpretations;

cause the given third-party agent to satisfy the spoken utterance based on the given interpretation of the spoken utterance

determine whether the given interpretation is one of the plurality of first-party interpretations or one of the plurality of third-party interpretations; and

in response to determining that the given interpretation is one of the plurality of first-party interpretations:

cause the automated assistant to provide, for presentation to the user of the client device, an indication that the given interpretation is one of the plurality of first-party interpretations; and

in response to determining that the given interpretation is one of the plurality of third-party interpretations:

cause the automated assistant to provide, for presentation to the user of the client device, an indication that the given interpretation is one of the plurality of third-party interpretations.

2. The system of claim 1 , wherein each of the plurality of first-party interpretations and each of the plurality of third-party interpretations include a corresponding intent, one or more corresponding parameters associated with the corresponding intent, and one or more corresponding slot values for each of the one or more corresponding parameters.

3. The system of claim 2 , wherein each of the plurality of first-party interpretations include one of: a unique corresponding intent, or one or more unique corresponding slot values for one or more of the corresponding parameters.

4. The system of claim 2 , wherein each of the plurality of third-party interpretations include one of: a unique corresponding intent, or one or more unique corresponding slot values for one or more of the corresponding parameters.

5. The system of claim 1 , wherein, in causing the automated assistant to provide, for presentation to the user of the client device, the indication that the given interpretation is one of the plurality of third-party interpretations, the at least one processor is operable to:

cause the automated assistant to audibly render the indication that the given interpretation is one of the plurality of third-party interpretations in a synthesized voice that is reserved for third-party agents, and that is distinct from an automated assistant voice reserved for the automated assistant.

6. A system comprising:

at least one processor; and

memory storing instructions that, when executed, cause the at least one processor to be operable to:

receive audio data that captures a spoken utterance of a user, the audio data being generated by one or more microphones of a client device of the user, and the spoken utterance being directed to an automated assistant executed at least in part at the client device;

determine, based on processing the audio data, a plurality of first-party interpretations of the spoken utterance, each of the plurality of first-party interpretations being associated with a corresponding first-party predicted value indicative of a magnitude of confidence that each of the first-party interpretations are predicted to satisfy the spoken utterance;

identify a given third-party agent capable of satisfying the spoken utterance;

transmit, to the given third-party agent and over one or more networks, and based on processing the audio data, one or more structured requests that, when received, causes the given third-party to determine a plurality of third-party interpretations of the spoken utterance, each of the plurality of third-party interpretations being associated with a corresponding third-party predicted value indicative of a magnitude of confidence that each of the third-party interpretations are predicted to satisfy the spoken utterance;

receive, from the given third-party agent and over one or more of the networks, the plurality of third-party interpretations of the spoken utterance;

select, based on the corresponding first-party predicted values and the corresponding third-party predicted values, a given interpretation of the spoken utterance from among the plurality of first-party interpretations and the plurality third-party interpretations; and

generate one or more training instances to be utilized in updating a machine learning (ML) model that is trained to select given interpretations for spoken utterances based on the plurality of first-party interpretations, the corresponding first-party predicted values, the plurality of third-party interpretations, and the corresponding third-party predicted values, wherein, in generating the one or more training instances to be utilized in updating the ML model, the at least one processor is operable to:

label the given interpretation of the spoken utterance that was selected with a positive training label; and

label one or more of the first-party interpretations and one or more of the third-party interpretations that were not selected with a corresponding negative training label.

7. The system of claim 6 , wherein the at least one processor is further operable to:

apply the plurality of first-party interpretations and the plurality of third-party interpretations as input across the ML model to generate predicted output associated with each the plurality of first-party interpretations and the plurality of third-party interpretations;

compare the predicted output associated with each the plurality of first-party interpretations and the plurality of third-party interpretations to the positive training label and the corresponding negative training labels to generate one or more losses; and

cause the ML model to be updated based on the one or more losses.

8. The system of claim 7 , wherein, in applying the plurality of first-party interpretations and the plurality of third-party interpretations as input across the ML model, the at least one processor is operable to:

apply the corresponding first-party predicted values and the corresponding third-party predicted values as input across the ML model to generate the predicted output.

9. The system of claim 7 , wherein, in applying the plurality of first-party interpretations and the plurality of third-party interpretations as input across the ML model further, the at least one processor is operable to:

apply an indication that data that is available to the given third party agent, but that is not available to the automated assistant, was utilized in generating the plurality of third-party interpretations as input across the ML model along with the corresponding first-party predicted values and the corresponding third-party predicted values.

10. The system of claim 7 , wherein, in applying the plurality of first-party interpretations and the plurality of third-party interpretations as input across the ML model further, the at least one processor is further operable to:

apply an indication of one or more intents and/or one or more corresponding parameters associated with the one or more intents as input across the ML model along with the corresponding first-party predicted values and the corresponding third-party predicted values.

11. The system of claim 10 , wherein the one or more intents and the one or more corresponding parameters are determined based on processing the audio data, wherein the indication of the one or more intents include an indication of the one or more intents are to be resolved in selecting the given interpretations, and wherein the indication of the one or more corresponding parameters include an indication of the one or more corresponding parameters are to be resolved in selecting the given interpretations.

12. The system of claim 6 , wherein, in labeling the given interpretation of the spoken utterance that was selected with a positive training label, the at least one processor is operable to:

determine whether one or more user satisfaction criteria are satisfied; and

wherein labeling the given interpretation of the spoken utterance that was selected with the positive training label is in response to determining that one or more of the user satisfaction criteria are satisfied.

13. A system comprising:

at least one processor implemented by an independent third-party agent; and

memory storing instructions that, when executed, cause the at least one processor implemented by the independent third-party agent to be operable to:

obtain a plurality of first-party interpretations of a spoken utterance, wherein each of the first-party interpretations of the spoken utterance are determined by an automated assistant, and wherein the automated assistant determines the first-party interpretations based on:

processing audio data that captures a spoken utterance of a user, the audio data being generated by one or more microphones of a client device of the user, and the spoken utterance being directed to the automated assistant executed at least in part at the client device;

determining, based on processing the audio data, a plurality of first-party interpretations of the spoken utterance, each of the plurality of first-party interpretations being associated with a corresponding first-party predicted value indicative of a magnitude of confidence that each of the first-party interpretations are predicted to satisfy the spoken utterance;

obtain a plurality of third-party interpretations of the spoken utterance, wherein each of the third-party interpretations of the spoken utterance are determined by one or more third-party agents, wherein the independent third-party agent excludes the automated assistant and the one or more third-party agents utilized in generating the plurality of third-party interpretations, and wherein each of the one or more third-party agents determine one or more of the third-party interpretations based on:

receiving, from the automated assistant over one or more networks, and based on processing the audio data, one or more structured requests; and

determining, based on the one or more structured requests, the plurality of third-party interpretations of the spoken utterance, each of the plurality of third-party interpretations being associated with a corresponding third-party predicted value indicative of a magnitude of confidence that each of the third-party interpretations are predicted to satisfy the spoken utterance;

select, based on the corresponding first-party predicted values and the corresponding third-party predicted values, a given interpretation of the spoken utterance from among the plurality of first-party interpretations and the plurality of third-party interpretations; and

transmit the given interpretation of the spoken utterance to the automated assistant or a given third-party agent, of the one or more third-party agents.

14. The system of claim 13 , wherein transmitting the given interpretation of the spoken utterance to the automated assistant causes the automated assistant to cause the given third-party agent to satisfy the spoken utterance based on the given interpretation of the spoken utterance.

15. The system of claim 13 , wherein transmitting the given interpretation of the spoken utterance to the given third-party agent causes the given third-party agent to satisfy the spoken utterance based on the given interpretation of the spoken utterance.

16. The system of claim 13 , wherein the at least one processor implemented by the independent third-party agent is further operable to:

determine whether to transmit the given interpretation of the spoken utterance to the automated assistant or the given third-party agent based on one or more predicted latency measures associated with transmitting the given interpretation of the spoken utterance to the automated assistant; and

in response to determining that the one or more predicted latency measures associated with transmitting the given interpretation of the spoken utterance to the automated assistant satisfy a latency threshold:

transmit the given interpretation of the spoken utterance to the given third-party agent.

17. The system of claim 16 , wherein the at least one processor implemented by the independent third-party agent is further operable to:

in response to determining that the one or more predicted latency measures associated with transmitting the given interpretation of the spoken utterance to the automated assistant fail to satisfy the latency threshold:

transmit the given interpretation of the spoken utterance to the automated assistant.

Assignments (1)

Continuity (3)

References Cited (15)