Granted Patent B2

US 12,400,634 · App. 17/726,244 · Granted Aug 26, 2025

Inventors: Martin Baeuml (Zurich, CH); Thushan Amarasiriwardena (Alameda, CA); Roberto Pieraccini (Zurich, CH); Gianluca Martini (Zurich, CH)

Assignee: GOOGLE LLC

G10L13/10G06F40/169G06T7/20G06V20/40G06V40/20G10L13/02G10L13/08G10L15/063G10L15/1815G10L15/183G10L15/22G10L25/57H04N5/04G06T2207/10016G06T2207/30196G10L2015/223

Abstract

Claims (64)

1. A method implemented by one or more processors, the method comprising:

receiving a stream of audio data that captures a spoken utterance of a user of a client device, the stream of audio data being generated by one or more microphones of the client device, and the spoken utterance being directed to an instance of an automated assistant executing at least in part at the client device;

generating, based on processing the stream of audio data, a given assistant output that is responsive to the spoken utterance, wherein the given assistant output includes: (i) a stream of textual content; and (ii) a stream of visual cues for controlling a display of the client device responsive to the spoken utterance and/or for controlling a visualized representation of the instance of the automated assistant that is visually rendered for presentation to the user via the display of the client device; and

modifying, based on a given assistant persona assigned to the instance of the automated assistant by the user and from among a plurality of disparate assistant personas, the given assistant output that is responsive to the spoken utterance to generate a modified given assistant output, wherein the modified given assistant output includes: (i) a stream of modified textual content that differs from the stream of textual content; and (ii) a stream of modified visual cues for controlling the display of the client device responsive to the spoken utterance and/or for controlling the visualized representation of the instance of the automated assistant that differs from the stream of visual cues;

annotating the stream of modified textual content with one or more visual cue timestamps that indicate when the stream of modified visual cues is to be utilized in controlling the display of the client device and/or for controlling the visualized representation of the instance of the automated assistant;

synchronizing, for presentation to the user, and based on the one or more visual cue timestamps, audible rendering of synthesized speech corresponding to the stream of modified textual content and the utilization of the stream of modified visual cues in controlling the display of the client device and/or in controlling the visualized representation of the instance of the automated assistant; and

in response to receiving the stream of audio data that captures the spoken utterance of the user of the client device:

causing synthesized speech audio data capturing the synthesized speech corresponding to the stream of modified textual content to be audibly rendered for presentation to the user via one or more speakers of the client device; and

causing the stream of modified visual cues to be utilized in controlling the display of the client device and/or in controlling the visualized representation of the instance of the automated assistant.

2. The method of claim 1 , wherein generating the given assistant output that is responsive to the spoken utterance based on processing the stream of audio data comprises:

processing, using an automatic speech recognition (ASR) model, the stream of audio data that captures the spoken utterance to generate a stream of ASR output;

processing, using a natural language understanding (NLU) model, the stream of ASR output to generate a stream of NLU output; and

determining, based on at least the stream of NLU output, the given assistant output that is responsive to the spoken utterance.

3. The method of claim 2 , wherein determining the given assistant output that is responsive to the spoken utterance based on the stream of NLU output comprises:

processing, using a large language model (LLM), the stream of ASR output and/or the stream of NLU output to determine the stream of textual content and the stream of visual cues included in the given assistant output.

4. The method of claim 2 , wherein determining the given assistant output that is responsive to the spoken utterance based on the stream of NLU output comprises:

processing, using large language model (LLM) output that was previously generated using an LLM based on a previous instance of the spoken utterance, the stream of ASR output and/or the stream of NLU output to determine the stream of textual content and the stream of visual cues included in the given assistant output.

5. The method of claim 2 , wherein determining the given assistant output that is responsive to the spoken utterance based on the stream of NLU output comprises:

generating, based on the stream of ASR output and/or the stream of NLU output, one or more structured requests;

transmitting, to one or more first-party agents and/or one or more third-party agents, the one or more structured requests; and

determining, based on content received responsive to the one or more structured requests, the stream of textual content and the stream of visual cues included in the given assistant output.

6. The method of claim 1 , wherein modifying the given assistant output that is responsive to the spoken output to generate the modified given assistant output based on the given assistant persona assigned to the instance of the automated assistant comprises:

obtaining, from one or more databases, given persona data that is specific to the given persona assigned to the instance of the automated assistant; and

processing the stream of textual content and the stream of visual cues along with the persona data that is specific to the given persona assigned to the instance of the automated assistant to generate the stream of modified textual content that differs from the stream of textual content and the stream of modified visual cues that differs from the stream of visual cues.

7. The method of claim 6 , wherein the persona data that is specific to the given persona assigned to the instance of the automated assistant includes a given persona token that is specific to the given persona assigned to the instance of the automated assistant and/or a given embedding that is specific to the given persona assigned to the instance of the automated assistant.

8. The method of claim 6 , wherein processing the stream of textual content and the stream of visual cues along with the persona data that is specific to the given persona assigned to the instance of the automated assistant to generate the stream of modified textual content that differs from the stream of textual content and the stream of modified visual cues that differs from the stream of visual cues comprises:

processing, using a large language model (LLM), the stream of textual content and the stream of visual cues along with the persona data that is specific to the given persona assigned to the instance of the automated assistant to generate the stream of modified textual content that differs from the stream of textual content and the stream of modified visual cues that differs from the stream of visual cues.

9. The method of claim 6 , wherein processing the stream of textual content and the stream of visual cues along with the persona data that is specific to the given persona assigned to the instance of the automated assistant to generate the stream of modified textual content that differs from the stream of textual content and the stream of modified visual cues that differs from the stream of visual cues comprises:

processing, using large language model (LLM) output that was previously generated using a LLM based on a previous instance of the spoken utterance, the stream of textual content and the stream of visual cues along with the persona data that is specific to the given persona assigned to the instance of the automated assistant to generate the stream of modified textual content that differs from the stream of textual content and the stream of modified visual cues that differs from the stream of visual cues.

10. The method of claim 1 , wherein the given persona assigned to the instance of the automated assistant is associated with:

a first vocabulary, of a plurality of disparate vocabularies, utilized in modifying the stream of textual content to generate the stream of modified textual content,

a first set of prosodic properties, of a plurality of disparate sets of prosodic properties, utilized in generating the synthesized speech audio data capturing the synthesized speech corresponding to the stream of modified textual content to be audibly rendered for presentation to the user, and/or

a first set of visual cues, of a plurality of disparate sets of visual cues, utilized in modifying the stream of visual cues to generate the stream of modified visual cues.

11. The method of claim 10 , wherein the stream of modified textual content that differs from the stream of textual content is modified using the first vocabulary, and wherein the stream of modified visual cues for controlling the display of the client device responsive to the spoken utterance and/or for controlling the visualized representation of the instance of the automated assistant that differs from the stream of visual cues is modified using the first set of visual cues.

12. The method of claim 11 , further comprising:

processing, using a text-to-speech (TTS) model and based on the first set of the prosodic properties, the stream of modified textual content to generate the synthesized speech audio data.

13. The method of claim 10 , further comprising:

receiving an additional stream of audio data that captures an additional spoken utterance of an additional user of an additional client device, the additional stream of audio data being generated by one or more additional microphones of the additional client device, the additional spoken utterance being directed to an additional instance of the automated assistant executing at least in part at the additional client device, and the additional spoken utterance being the same as the spoken utterance;

generating, based on processing the stream of additional audio data, the given assistant output that is responsive to the additional spoken utterance, wherein the given assistant output includes: (i) the stream of textual content; and (ii) the stream of visual cues for controlling an additional display of the additional client device responsive to the additional spoken utterance and/or for controlling an additional visualized representation of the additional instance of the automated assistant that is visually rendered for presentation to the additional user via the additional display of the additional client device; and

modifying, based on a given additional assistant persona assigned to the additional instance of the automated assistant that is in addition to the given assistant persona by the additional user and from among the plurality of disparate assistant personas, the given assistant output that is responsive to the additional spoken utterance to generate a modified given additional assistant output, wherein the modified given additional assistant output includes: (i) a stream of modified additional textual content that differs from the stream of textual content and that differs from the stream of modified textual content; and (ii) a stream of modified additional visual cues for controlling the additional display of the additional client device responsive to the additional spoken utterance and/or for controlling the additional visualized representation of the additional instance of the automated assistant that differs from the stream of visual cues and that differs from the stream of modified visual cues; and

in response to receiving the additional stream of audio data that captures the additional spoken utterance of the additional user of the additional client device:

causing additional synthesized speech audio data capturing additional synthesized speech corresponding to the stream of modified additional textual content to be audibly rendered for presentation to the additional user via one or more additional speakers of the additional client device; and

causing the stream of modified additional visual cues to be utilized in controlling the additional display of the additional client device and/or in controlling the additional visualized representation of the additional instance of the automated assistant.

14. The method of claim 1 , wherein the stream of modified visual cues is utilized in controlling the display of the client device responsive to the spoken utterance, and wherein the stream of modified visual cues that are utilized in controlling the display of the client device responsive to the spoken utterance include one or more display animations that cause the display of the client device to be dynamically adapted while the synthesized speech audio data is being audibly rendered for presentation to the user.

15. The method of claim 1 , wherein the stream of modified visual cues is utilized in controlling the visualized representation of the instance of the automated assistant, and wherein the stream of modified visual cues that are utilized in controlling the visualized representation of the instance of the automated assistant include one or more animated physical gesture motions performed by the visualized representation of the instance of the automated assistant while the synthesized speech audio data is being audibly rendered for presentation to the user.

16. The method of claim 1 , wherein the one or more visual cue timestamps include at least a start visual cue timestamp that indicates when a given visual cue, included in the stream of modified visual cues, will start being utilized in controlling the display of the client device and/or for controlling the visualized representation of the instance of the automated assistant, and a stop visual cue timestamp that indicates when the given visual cue, included in the stream of modified visual cues, will stop being utilized in controlling the display of the client device and/or for controlling the visualized representation of the instance of the automated assistant.

17. A method implemented by one or more processors, the method comprising:

receiving a stream of audio data that captures a spoken utterance of a user of a client device, the stream of audio data being generated by one or more microphones of the client device, and the spoken utterance being directed to an instance of an automated assistant executing at least in part at the client device;

generating, based on processing the stream of audio data and using a given large language model (LLM), a given assistant output that is responsive to the spoken utterance and that is specific to a given persona assigned to the instance of the automated assistant from among a plurality of disparate personas, wherein the given assistant output includes: (i) a stream of textual content that is specific to the given persona of the instance of the automated assistant; and (ii) a stream of visual cues for controlling a display of the client device responsive to the spoken utterance and/or for controlling a visualized representation of the instance of the automated assistant that is visually rendered for presentation to the user via the display of the client device and that is specific to the given persona assigned to the instance of the automated assistant;

annotating the stream of textual content with one or more visual cue timestamps that indicate when the stream of visual cues is to be utilized in controlling the display of the client device and/or for controlling the visualized representation of the instance of the automated assistant;

synchronizing, for presentation to the user, and based on the one or more visual cue timestamps, audible rendering of synthesized speech corresponding to the stream of textual content and the utilization of the stream of visual cues in controlling the display of the client device and/or in controlling the visualized representation of the instance of the automated assistant; and

in response to receiving the stream of audio data that captures the spoken utterance of the user of the client device:

causing synthesized speech audio data capturing the synthesized speech corresponding to the stream of textual content to be audibly rendered for presentation to the user via one or more speakers of the client device; and

causing the stream of visual cues to be utilized in controlling the display of the client device and/or in controlling the visualized representation of the instance of the automated assistant.

18. A system comprising:

at least one processor; and

memory storing instructions that, when executed, cause the at least one processor to be operable to:

receive a stream of audio data that captures a spoken utterance of a user of a client device, the stream of audio data being generated by one or more microphones of the client device, and the spoken utterance being directed to an instance of an automated assistant executing at least in part at the client device;

generate, based on processing the stream of audio data and using a given large language model (LLM), a given assistant output that is responsive to the spoken utterance and that is specific to a given persona assigned to the instance of the automated assistant from among a plurality of disparate personas, wherein the given assistant output includes: (i) a stream of textual content that is specific to the given persona of the instance of the automated assistant; and (ii) a stream of visual cues for controlling a display of the client device responsive to the spoken utterance and/or for controlling a visualized representation of the instance of the automated assistant that is visually rendered for presentation to the user via the display of the client device and that is specific to the given persona assigned to the instance of the automated assistant;

annotate the stream of textual content with one or more visual cue timestamps that indicate when the stream of visual cues is to be utilized in controlling the display of the client device and/or for controlling the visualized representation of the instance of the automated assistant;

synchronize, for presentation to the user, and based on the one or more visual cue timestamps, audible rendering of synthesized speech corresponding to the stream of textual content and the utilization of the stream of visual cues in controlling the display of the client device and/or in controlling the visualized representation of the instance of the automated assistant; and

in response to receiving the stream of audio data that captures the spoken utterance of the user of the client device:

cause synthesized speech audio data capturing the synthesized speech corresponding to the stream of textual content to be audibly rendered for presentation to the user via one or more speakers of the client device; and

cause the stream of visual cues to be utilized in controlling the display of the client device and/or in controlling the visualized representation of the instance of the automated assistant.

Assignments (1)

Continuity (1)

References Cited (16)

Cited By (1)

US 12,737,201