Granted Patent B2

US 12,334,049 · App. 18/074,799 · Granted Jun 17, 2025

Inventors: Asaf Aharoni (Ramat Hasharon, IL); Eyal Segalis (Tel Aviv, IL); Sasha Goldshtein (Tel Aviv, IL); Ofer Ron (Givatayim, IL); Yaniv Leviathan (New York, NY); Yoav Tzur (Tel Aviv, IL)

Assignee: GOOGLE LLC

G10L13/08G10L15/22G10L2015/221G10L2015/223

Abstract

Claims (77)

1. A method implemented by one or more processors of a client device, the method comprising:

receiving, at the client device, unstructured free-form natural language input from a user of the client device, the unstructured free-form natural language input including one or more tasks associated with an entity;

in response to receiving the unstructured free-form natural language input that includes the one or more tasks associated with the entity:

generating, based on at least the unstructured free-form natural language input and locally at the client device, a chatbot to perform the one or more tasks associated with the entity and on behalf of the user; and

causing the chatbot to perform the one or more tasks associated with the entity and on behalf of the user, wherein causing the chatbot to perform the one or more tasks associated with the entity and on behalf of the user comprises:

causing the chatbot to engage in a corresponding conversation with the entity;

during the corresponding conversation with the entity:

causing the chatbot to render a plurality of instances of synthesized speech audio data for presentation to a representative of the entity, wherein at least a given instance of synthesized speech audio data, from among the plurality of instances of synthesized speech audio data, conveys details of the one or more tasks associated with the entity; and

receiving responsive content that is responsive to at least the given instance of synthesized speech audio data; and

causing the responsive content to be provided for presentation to the user of the client device.

2. The method of claim 1 , wherein generating the chatbot to perform the one or more tasks associated with the entity and on behalf of the user comprises:

obtaining a previously trained large language model (LLM);

causing the previously trained LLM to be fine-tuned based on the unstructured free-form natural language input to generate a fine-tuned LLM; and

utilizing the fine-tuned LLM as the chatbot.

3. The method of claim 2 , wherein causing the chatbot to render the given instance of synthesized speech audio data for presentation to the representative comprises:

processing, using the fine-tuned LLM, one or more features of the unstructured free-form natural language input to generate a given instance of textual data that conveys the details of the one or more tasks associated with the entity;

processing, using a text-to-speech (TTS) model, the given instance of textual data that conveys the details of the one or more tasks associated with the entity to generate the given instance of synthesized speech audio data; and

transmitting, from the client device and to an additional client device of the representative, the given instance of synthesized speech audio data, wherein transmitting the given instance of synthesized speech audio data to the additional client device causes the additional client device to audibly render the given instance of synthesized speech audio data for presentation to the representative via one or more speakers of the additional client device.

4. The method of claim 3 , further comprising:

processing, using the fine-tuned LLM, and along with one or more of the features of the unstructured free-form natural language input, a corresponding context of the corresponding conversation to generate the given instance of textual data that conveys the details of the one or more tasks associated with the entity.

5. The method of claim 3 , further comprising:

in response to the given instance of synthesized speech audio data being audibly rendered for presentation to the representative via the one or more speakers of the additional client device:

receiving, at the client device and from the additional client device, a given instance of response audio data that includes the responsive content that is responsive to at least the given instance of synthesized speech audio data;

processing, using an automatic speech recognition (ASR) model, the given instance of response audio data to generate a given instance of response textual data; and

determining, based on the given instance of response textual data, whether performance of one or more of the tasks is successfully completed during the corresponding conversation.

6. The method of claim 5 , wherein causing the responsive content to be provided for presentation to the user of the client device is in response to determining that one or more of the tasks is successfully completed during the corresponding conversation.

7. The method of claim 5 , further comprising:

in response to determining that one or more of the tasks is not successfully completed during the corresponding conversation:

causing the chatbot to render an indication that the user is being prompted to join the corresponding conversation for presentation to the representative of the entity;

generating a prompt that requests the user to join the corresponding conversation; and

causing the prompt that requests the user to join the corresponding conversation to be provided for presentation to the user at the client device.

8. The method of claim 7 , wherein the prompt further includes a certain reason with respect to why one or more of the tasks is not successfully completed during the corresponding conversation.

9. The method of claim 3 , further comprising:

prior to processing one or more of the features of the unstructured free-form natural language input using the fine-tuned LLM:

extracting one or more of the features from the unstructured free-form natural language input.

10. The method of claim 9 , wherein one or more of the features are explicitly included in the unstructured free-form natural language input, and wherein extracting one or more of the features from the unstructured free-form natural language input that are explicitly included in the unstructured free-form natural language input comprises:

utilizing an input parser to extract one or more of the features are explicitly included in the unstructured free-form natural language input.

11. The method of claim 9 , wherein one or more of the features are implicitly included in the unstructured free-form natural language input, and wherein extracting one or more of the features from the unstructured free-form natural language input that are implicitly included in the unstructured free-form natural language input comprises:

utilizing an input parser to identify one or more of the features are implicitly included in the unstructured free-form natural language input; and

utilizing a coreference resolver to extract one or more of the features are implicitly included in the unstructured free-form natural language input.

12. The method of claim 2 , wherein the previously trained LLM is stored in on-device storage of the client device, and wherein the previously trained LLM that is stored in the on-device storage of the client device is a sparsified version of a global previously trained LLM that is available at a remote system communicatively coupled to the client device.

13. The method of claim 1 , further comprising:

subsequent to causing the chatbot to perform the one or more tasks associated with the entity and on behalf of the user:

discarding the fine-tuned LLM from the on-device storage of the client device; and

refraining from discarding the previously trained LLM from the on-device storage of the client device.

14. The method of claim 1 , further comprising:

identifying, based on the unstructured free-form natural language input, the entity that is associated with the one or more tasks;

determining a corresponding identifier for the entity that is associated with the one or more tasks; and

utilizing the corresponding identifier for the entity that is associated with the one or more tasks to cause the chatbot to engage in the corresponding conversation with the entity.

15. The method of claim 14 , wherein the corresponding identifier for the entity that is associated with the one or more tasks is a corresponding telephone number for the entity, and wherein utilizing the corresponding identifier for the entity that is associated with the one or more tasks to cause the chatbot to engage in the corresponding conversation with the entity comprises:

causing the chatbot to utilize the corresponding telephone number for the entity to initiate an automated telephone call, that is automated on behalf of the user, to perform the one or more tasks associated with the entity.

16. A method implemented by one or more processors of a remote system, the method comprising:

receiving, at the remote system and from a client device, an indication of unstructured free-form natural language input from a user of the client device, the unstructured free-form natural language input including one or more tasks associated with an entity;

in response to receiving the indication of the unstructured free-form natural language input that includes the one or more tasks associated with an entity:

generating, based on at least the indication of the natural language input and remotely at the remote system, a chatbot to perform the one or more tasks associated with the entity and on behalf of the user; and

causing the chatbot to perform the one or more tasks associated with the entity and on behalf of the user, wherein causing the chatbot to perform the one or more tasks associated with the entity and on behalf of the user comprises:

causing the chatbot to engage in a corresponding conversation with the entity;

during the corresponding conversation with the entity:

causing the chatbot to render a plurality of instances of synthesized speech audio data for presentation to a representative of the entity, wherein at least a given instance of synthesized speech audio data, from among the plurality of instances of synthesized speech audio data, conveys details of the one or more tasks associated with the entity; and

receiving responsive content that is responsive to at least the given instance of synthesized speech audio data; and

transmitting, from the remote system and to the client device, an indication of the responsive content, wherein transmitting the indication of the responsive content to the client device causes the client device to provide the responsive content for presentation to the user of the client device.

17. The method of claim 16 , wherein generating the chatbot to perform the one or more tasks associated with the entity and on behalf of the user comprises:

obtaining a previously trained large language model (LLM);

causing the previously trained LLM to be fine-tuned based on the unstructured free-form natural language input to generate a fine-tuned LLM; and

utilizing the fine-tuned LLM as the chatbot.

18. The method of claim 17 , wherein the previously trained LLM is stored in remote storage of the remote system, and wherein the previously trained LLM that is stored in the remote storage of the remote system is an unsparsified version of a global previously trained LLM that is available at the remote system communicatively coupled to the client device.

19. The method of claim 17 , wherein the fine-tuned LLM is stored in the remote storage of the remote system.

20. A method implemented by one or more processors of a client device, the method comprising:

receiving, at the client device, unstructured free-form natural language input from a user of the client device, the unstructured free-form natural language input including one or more tasks associated with an entity;

in response to receiving the indication of the unstructured free-form natural language input that includes the one or more tasks associated with an entity:

generating, based on at least an indication of the natural language input and locally at the client device, a chatbot to perform the one or more tasks associated with the entity and on behalf of the user; and

causing the chatbot to perform the one or more tasks associated with the entity and on behalf of the user, wherein causing the chatbot to perform the one or more tasks associated with the entity and on behalf of the user comprises:

causing the chatbot to engage in a corresponding conversation with the entity;

during the corresponding conversation with the entity:

causing the chatbot to render a plurality of instances of textual data for presentation to a representative of the entity, wherein at least a given instance of textual data, from among the plurality of instances of textual data, conveys details of the one or more tasks associated with the entity; and

receiving responsive content that is responsive to at least the given instance of textual data; and

causing the responsive content to be provided for presentation to the user of the client device.

Assignments (1)

Continuity (1)

References Cited (49)

Cited By (1)

US 12,505,831