Granted Patent B2

US 12,469,498 · App. 18/594,833 · Granted Nov 11, 2025

Inventors: Quan Wang (Hoboken, NJ); Prashant Sridhar (New York, NY); Ignacio Lopez Moreno (New York, NY); Hannah Muckenhim (Martigny, CH)

Assignee: GOOGLE LLC

G10L17/04G10L17/00G10L17/02G10L17/18G10L17/22G10L25/18

Abstract

Claims (87)

1 . A client device comprising:

one or more processors, and

memory configured to store instructions that, when executed by the one or more processors, cause the one or more processors to perform a method that includes:

invoking an automated assistant client at the client device, wherein invoking the automated assistant client is in response to detecting one or more instances of invocation user interface input;

in response to invoking the automated assistant client:

capturing one or more instances of spoken input via one or more microphones of the client device;

performing certain processing of the one or more instances of spoken input;

generating a responsive action based on the certain processing of the one or more instances of spoken input;

causing performance of the responsive action;

determining that a continued listening mode is activated for the automated assistant client device;

in response to the continued listening mode being activated:

automatically monitoring for additional spoken input including after causing performance of at least part of the responsive action;

receiving audio data during the automatically monitoring;

determining whether the audio data includes any additional spoken input that is from a same human speaker who provided the spoken input, wherein determining whether the audio data includes the additional spoken input that is from the same human speaker comprises:

 identifying a speaker embedding for the human speaker that provided the one or more instances of spoken input, wherein the speaker embedding was generated based on one or more respective instances of speaker audio data corresponding to the human speaker;

 generating a refined version of the audio data, based on the speaker embedding and the one or more instances of spoken input, that isolates any of the audio data that is from the human speaker; and

 determining, based on the refined audio data, whether the audio data includes any additional spoken input that is from the same human.

2 . The client device of claim 1 , wherein the one or more instances of invocation of the user interface input is a selection of a graphical user interface element of the client device.

3 . The client device of claim 1 , wherein determining, based on the refined audio data, whether the audio data includes any additional spoken input that is from the same human comprises:

processing the refined version of the audio data to determine whether the refined version of the audio data includes one or more non-null segments; and

in response to determining the refined version of the audio data includes one or more non-null segments, determining the refined audio data includes any additional spoken input that is from the same human.

4 . The client device of claim 3 , in response to determining the audio data includes any additional spoken input that is from the same human speaker that provided the spoken input and further comprising:

performing one or more additional actions that are based on the refined version of the audio data.

5 . The client device of claim 4 , wherein performing the one or more additional actions that are based on the refined version of the audio data comprises:

generating additional responsive content that is customized for the human speaker and that is based on the refined version of the audio data; and

causing the client device to render output based on the additional responsive content.

6 . The client device of claim 1 , wherein determining, based on the refined audio data, whether the audio data includes any additional spoken input that is from the same human comprises:

processing the refined version of the audio data to determine whether the refined version of the audio data includes one or more non-null segments; and

in response to determining the refined version of the audio data does not include one or more non-null segments, determining the audio data does not include any additional spoken input that is from the same human.

7 . The client device of claim 6 , in response to determining the audio data does not include any additional spoken input that is from the same human that provided the spoken input and further comprising:

performing one or more additional actions that are based on the audio data capturing additional spoken input.

8 . The client device of claim 7 , wherein performing one or more additional actions that are based on the audio data capturing the additional spoken input comprises:

generating additional responsive content that is not customized for the human speaker; and

causing the client device to render output based on the additional responsive content.

9 . A method implemented by one or more processors, the method comprising:

invoking an automated assistant client at a client device, wherein invoking the automated assistant client is in response to detecting one or more instances of invocation user interface input;

in response to invoking the automated assistant client:

capturing one or more instances of spoken input via one or more microphones of the client device;

performing certain processing of the one or more instances of spoken input;

generating a responsive action based on the certain processing of the one or more instances of spoken input;

causing performance of the responsive action;

determining that a continued listening mode is activated for the automated assistant client device;

in response to the continued listening mode being activated:

automatically monitoring for additional spoken input including after causing performance of at least part of the responsive action;

receiving audio data during the automatically monitoring;

determining whether the audio data includes any additional spoken input that is from a same human speaker who provided the spoken input, wherein determining whether the audio data includes the additional spoken input that is from the same human speaker comprises:

 identifying a speaker embedding for the human speaker that provided the one or more instances of spoken input, wherein the speaker embedding was generated based on one or more respective instances of speaker audio data corresponding to the human speaker;

 generating a refined version of the audio data, based on the speaker embedding and the one or more instances of spoken input, that isolates any of the audio data that is from the human speaker; and

 determining, based on the refined audio data, whether the audio data includes any additional spoken input that is from the same human.

10 . The method of claim 9 , wherein the one or more instances of invocation of the user interface input is a selection of a graphical user interface element of the client device.

11 . The method of claim 9 , wherein determining, based on the refined audio data, whether the audio data includes any additional spoken input that is from the same human comprises:

processing the refined version of the audio data to determine whether the refined version of the audio data includes one or more non-null segments; and

in response to determining the refined version of the audio data includes one or more non-null segments, determining the refined audio data includes any additional spoken input that is from the same human.

12 . The method of claim 11 , in response to determining the audio data includes any additional spoken input that is from the same human speaker that provided the spoken input and further comprising:

performing one or more additional actions that are based on the refined version of the audio data.

13 . The method of claim 12 , wherein performing the one or more additional actions that are based on the refined version of the audio data comprises:

generating additional responsive content that is customized for the human speaker and that is based on the refined version of the audio data; and

causing the client device to render output based on the additional responsive content.

14 . The method of claim 9 , wherein determining, based on the refined audio data, whether the audio data includes any additional spoken input that is from the same human comprises:

processing the refined version of the audio data to determine whether the refined version of the audio data includes one or more non-null segments; and

in response to determining the refined version of the audio data does not include one or more non-null segments, determining the audio data does not include any additional spoken input that is from the same human.

15 . The method of claim 14 , in response to determining the audio data does not include any additional spoken input that is from the same human that provided the spoken input and further comprising:

performing one or more additional actions that are based on the audio data capturing additional spoken input.

16 . The method of claim 15 , wherein performing one or more additional actions that are based on the audio data capturing the additional spoken input comprises:

generating additional responsive content that is not customized for the human speaker; and

causing the client device to render output based on the additional responsive content.

17 . A non-transitory computer-readable storage medium storing instructions executable by one or more processors of a client device to perform a method of:

invoking an automated assistant client at the client device, wherein invoking the automated assistant client is in response to detecting one or more instances of invocation user interface input;

in response to invoking the automated assistant client:

capturing one or more instances of spoken input via one or more microphones of the client device;

performing certain processing of the one or more instances of spoken input;

generating a responsive action based on the certain processing of the one or more instances of spoken input;

causing performance of the responsive action;

determining that a continued listening mode is activated for the automated assistant client device;

in response to the continued listening mode being activated:

automatically monitoring for additional spoken input including after causing performance of at least part of the responsive action;

receiving audio data during the automatically monitoring;

determining whether the audio data includes any additional spoken input that is from a same human speaker who provided the spoken input, wherein determining whether the audio data includes the additional spoken input that is from the same human speaker comprises:

identifying a speaker embedding for the human speaker that provided the one or more instances of spoken input, wherein the speaker embedding was generated based on one or more respective instances of speaker audio data corresponding to the human speaker;

generating a refined version of the audio data, based on the speaker embedding and the one or more instances of spoken input, that isolates any of the audio data that is from the human speaker; and

determining, based on the refined audio data, whether the audio data includes any additional spoken input that is from the same human.

18 . The non-transitory computer-readable storage medium of claim 17 , wherein the one or more instances of invocation of the user interface input is a selection of a graphical user interface element of the client device.

19 . The non-transitory computer-readable storage medium of claim 17 , wherein determining, based on the refined audio data, whether the audio data includes any additional spoken input that is from the same human comprises:

processing the refined version of the audio data to determine whether the refined version of the audio data includes one or more non-null segments; and

in response to determining the refined version of the audio data includes one or more non-null segments, determining the refined audio data includes any additional spoken input that is from the same human.

20 . The non-transitory computer-readable storage medium of claim 19 , in response to determining the audio data includes any additional spoken input that is from the same human speaker that provided the spoken input and further comprising:

performing one or more additional actions that are based on the refined version of the audio data.

Assignments (1)

Continuity (4)

References Cited (57)