Granted Patent B2

US 12,283,270 · App. 17/541,098 · Granted Apr 22, 2025

Inventors: Asaf Aharoni (Ramat Hasharon, IL); Yaniv Leviathan (New York, NY); Eyal Segalis (Tel Aviv, IL); Gal Elidan (Modiin, IL); Sasha Goldshtein (Tel Aviv, IL); Tomer Amiaz (Tel Aviv, IL); Deborah Cohen (Tel Aviv, IL)

Assignee: GOOGLE LLC

G10L15/063G10L15/22G10L15/30H04L67/133H04L67/53H04M3/5166G10L2015/0635G10L2015/228

Abstract

Claims (73)

1. A method implemented by one or more processors, the method comprising:

obtaining, via a voice bot development platform, a plurality of remote procedure call (RPC) outbound training instances, each of the plurality of RPC outbound training instances including:

training instance input, the training instance input including at least a portion of a corresponding conversation and a prior context of the corresponding conversation, and

training instance output, the training instance output including a corresponding ground truth response to at least the portion of the corresponding conversation, wherein the corresponding ground truth response for the training instance output of a given RPC outbound training instance, of the plurality of RPC outbound training instances, comprises at least a corresponding RPC outbound request;

training, via the voice bot development platform, a voice bot based on at least the plurality of RPC outbound training instances,

wherein training the voice bot based on the plurality of RPC outbound training instances causes the voice bot to interact with a third-party system,

wherein training the voice bot based on the given RPC outbound training instance comprises utilizing one or more attention mechanisms to attention the voice bot to generate the corresponding RPC outbound request, to be transmitted to the third-party system, and based on processing at least the portion of the corresponding conversation and the prior context of the corresponding conversation, and

wherein processing at least the portion of the corresponding conversation and the prior context of the corresponding conversation to train the voice bot comprises:

processing, using a plurality of machine learning (ML) layers of a ML model, and for the given RPC outbound training instance, at least the portion of the corresponding conversation and the prior context of the corresponding conversation to generate an embedding associated with a current state of the corresponding conversation;

processing, using a plurality of additional ML layers of the ML model, at least the embedding associated with the current state of the corresponding conversation to generate a predicted embedding associated with a predicted response to at least the portion of the corresponding conversation;

comparing, in embedding space, the predicted embedding associated with the predicted response to at least the portion of the corresponding conversation and a corresponding ground truth embedding associated with the corresponding RPC outbound request;

generating, based on comparing the predicted embedding and the corresponding ground truth embedding, one or more losses; and

updating the ML model based on one or more of the losses; and

subsequent to training the voice bot:

causing the trained voice bot to be deployed for conducting conversations on behalf of a third-party.

2. The method of claim 1 , wherein the portion of the corresponding conversation comprises a plurality of speech hypotheses for at least the portion of the corresponding conversation, and wherein processing at least the portion of the corresponding conversation and the prior context of the corresponding conversation to generate the embedding associated with the current state of the corresponding conversation comprises:

processing, using first ML layers of the plurality of ML layers, the plurality of speech hypotheses to generate a first embedding,

processing, using second ML layers of the plurality of ML layers, the prior context of the corresponding conversation to generate a second embedding, and

concatenating the first embedding and the second embedding to generate the embedding associated with the current state of the corresponding conversation.

3. The method of claim 2 , further comprising:

generating, via the voice bot development platform, a plurality of affinity features based on the embedding associated with the current state of the corresponding conversation.

4. The method of claim 3 , wherein processing at least the embedding associated with the current state of the corresponding conversation to generate the predicted embedding associated with the predicted response to at least the portion of the corresponding conversation and using the plurality of additional ML layers of the ML model comprises:

processing, using the plurality of additional ML layers of the ML model, and along with the embedding associated with the current state of the corresponding conversation, the plurality of affinity features to generate the predicted embedding associated with the predicted response to at least the portion of the corresponding conversation.

5. The method of claim 1 , wherein the third-party that deploys the voice bot is distinct from an additional third-party associated with the third-party system.

6. The method of claim 1 , wherein causing the trained voice bot to be deployed for conducting the conversations on behalf of the third-party comprises causing the trained voice bot to be deployed for conducting the conversations for telephone calls associated with the third-party, and wherein causing the trained voice bot to be deployed for conducting the conversations for the telephone calls associated with the third-party comprises:

causing the voice bot to answer corresponding incoming telephone calls and to conduct the conversations with corresponding humans that initiated the corresponding incoming telephone calls via respective client devices, wherein the voice bot, during the conversations with the corresponding humans, interacts with the third-party system via a corresponding RPC.

7. The method of claim 6 , further comprising:

subsequent to concluding the incoming telephone call:

generating a corresponding conversation summary for each of the conversations conducted during the corresponding incoming telephone calls, wherein one or more of the corresponding conversation summaries include an indication that the corresponding RPC occurred based on corresponding spoken utterances of the corresponding humans received during the corresponding incoming telephone calls; and

causing the corresponding conversation summaries of the conversations to be rendered via the voice bot development platform.

8. A method implemented by one or more processors, the method comprising:

obtaining, via a voice bot development platform, a plurality of remote procedure call (RPC) inbound training instances, each of the plurality of RPC inbound training instances including:

training instance input, the training instance input including at least a portion of a corresponding conversation and a prior context of the corresponding conversation, wherein at least the portion of the corresponding conversation for the training instance input of a given RPC inbound training instance, of the plurality of RPC inbound training instances, comprises at least a corresponding RPC inbound request, and

training instance output, the training instance output including a corresponding ground truth response to at least the portion of the corresponding conversation;

training, via the voice bot development platform, a voice bot based on at least the plurality of RPC inbound training instances,

wherein training the voice bot based on the plurality of RPC inbound training instances causes the voice bot to interact with a third-party system,

wherein training the voice bot based on the given RPC inbound training instance comprises utilizing one or more attention mechanisms to attention the voice bot to receive the corresponding RPC inbound request, from the third-party system, and based on processing at least the portion of the corresponding conversation and the prior context of the corresponding conversation, and

wherein processing at least the portion of the corresponding conversation and the prior context of the corresponding conversation to train the voice bot comprises:

processing, using a plurality of machine learning (ML) layers of a ML model, and for the given RPC inbound training instance, at least the portion of the corresponding conversation and the prior context of the corresponding conversation to generate an embedding associated with a current state of the corresponding conversation;

processing, using a plurality of additional ML layers of the ML model, at least the embedding associated with the current state of the corresponding conversation to generate a predicted embedding associated with a predicted response to at least the portion of the corresponding conversation;

comparing, in embedding space, the predicted embedding associated with the predicted response to at least the portion of the corresponding conversation and a corresponding ground truth embedding associated with the corresponding RPC inbound request;

generating, based on comparing the predicted embedding and the corresponding ground truth embedding, one or more losses; and

updating the ML model based on one or more of the losses; and

subsequent to training the voice bot:

causing the trained voice bot to be deployed for conducting conversations on behalf of a third-party.

9. The method of claim 8 , wherein processing at least the portion of the corresponding conversation and the prior context of the corresponding conversation to generate the embedding associated with the current state of the corresponding conversation comprises:

processing, using first ML layers of the plurality of ML layers, at least the RPC inbound request to generate a first embedding,

processing, using second ML layers of the plurality of ML layers, the prior context of the corresponding conversation to generate a second embedding, and

concatenating the first embedding and the second embedding to generate a predicted embedding associated with a predicted response to at least the portion of the corresponding conversation.

10. The method of claim 8 , wherein the third-party that deploys the voice bot is distinct from an additional third-party associated with the third-party system.

11. The method of claim 8 , wherein causing the trained voice bot to be deployed for conducting the conversations on behalf of the third-party comprises causing the trained voice bot to be deployed for conducting the conversations for telephone calls associated with the third-party, and wherein causing the trained voice bot to be deployed for conducting the conversations for the telephone calls associated with the third-party comprises:

causing the voice bot to answer corresponding incoming telephone calls and to conduct the conversations with corresponding humans that initiated the corresponding incoming telephone calls via respective client devices, wherein the voice bot, during the conversations with the corresponding humans, interacts with the third-party system via a corresponding RPC.

12. The method of claim 11 , further comprising:

subsequent to concluding the incoming telephone call:

generating a corresponding conversation summary for each of the conversations conducted during the corresponding incoming telephone calls, wherein one or more of the corresponding conversation summaries include an indication that the corresponding RPC occurred based on corresponding spoken utterances of the corresponding humans received during the corresponding incoming telephone calls; and

causing the corresponding conversation summaries of the conversations to be rendered via the voice bot development platform.

13. A system comprising:

at least one processor; and

memory storing instructions that, when executed, cause the at least one processor to be operable to;

obtain, via a voice bot development platform, a plurality of remote procedure call (RPC) outbound training instances, each of the plurality of RPC outbound training instances including:

training instance input, the training instance input including at least a portion of a corresponding conversation and a prior context of the corresponding conversation, and

training instance output, the training instance output including a corresponding ground truth response to at least the portion of the corresponding conversation, wherein the corresponding ground truth response for the training instance output of a given RPC outbound training instance, of the plurality of RPC outbound training instances, comprises at least a corresponding RPC outbound request;

train, via the voice bot development platform, a voice bot based on at least the plurality of RPC outbound training instances,

wherein training the voice bot based on the plurality of RPC outbound training instances causes the voice bot to interact with a third-party system,

wherein, in training the voice bot based on the given RPC outbound training instance, the at least one processor is operable to utilize one or more attention mechanisms to attention the voice bot to generate the corresponding RPC outbound request, to be transmitted to the third-party system, and based on processing at least the portion of the corresponding conversation and the prior context of the corresponding conversation, and

and

wherein, in processing at least the portion of the corresponding conversation and the prior context of the corresponding conversation to train the voice bot, the at least one processor is operable to:

process, using a plurality of machine learning (ML) layers of a ML model, and for the given RPC outbound training instance, at least the portion of the corresponding conversation and the prior context of the corresponding conversation to generate an embedding associated with a current state of the corresponding conversation;

process, using a plurality of additional ML layers of the ML model, at least the embedding associated with the current state of the corresponding conversation to generate a predicted embedding associated with a predicted response to at least the portion of the corresponding conversation;

compare, in embedding space, the predicted embedding associated with the predicted response to at least the portion of the corresponding conversation and a corresponding ground truth embedding associated with the corresponding RPC outbound request;

generate, based on comparing the predicted embedding and the corresponding ground truth embedding, one or more losses; and

update the ML model based on one or more of the losses; and subsequent to training the voice bot:

cause the trained voice bot to be deployed for conducting conversations on behalf of a third-party.

Assignments (1)

Continuity (2)

References Cited (42)