Granted Patent B2

US 12,334,074 · App. 18/606,066 · Granted Jun 17, 2025

Inventors: Robert A. Zurek (Antioch, IL); Adrian M. Schuster (West Olive, MI); Fu-Lin Shau (Lake Zurich, IL); Jincheng Wu (Naperville, IL)

Assignee: Google Technology Holdings LLC

G10L15/22G06F3/013G06V20/59G06V40/166G06V40/19G06V40/20G10L15/20G10L15/25G10L15/26G10L21/0208G06V40/18G10L2015/223G10L2015/227G10L15/24G10L2021/02166G10L25/78H04R2430/20H04R2460/07H04R2499/11

Abstract

Claims (62)

1. A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:

obtaining image data comprising a representation of a first user and a second user;

obtaining audio data comprising:

a first voice data corresponding to the first user speaking; and

a second voice data corresponding to the second user speaking;

associating, based on the image data, the first voice data to a first voice-recognition database of the first user speaking and the second voice data to a second voice-recognition database of the second user speaking;

generating, using speech-to-text conversion, a transcription of the audio data;

annotating, based on the first voice data associated with the first voice-recognition database, a first portion of the transcription corresponding to the first voice data with a first annotation identifying the first user; and

annotating, based on the second voice data associated with the second voice-recognition database, a second portion of the transcription corresponding to the second voice data with a second annotation identifying the second user.

2. The computer-implemented method of claim 1 , wherein the operations further comprise:

determining a direction of the first user relative to a computing device that captured the image data; and

adjusting, based on the direction of the first user relative to the computing device that captured the image data, a microphone beamform.

3. The computer-implemented method of claim 2 , wherein adjusting the microphone beamform comprises adjusting the microphone beamform to better isolate and capture the first voice data.

4. The computer-implemented method of claim 1 , wherein the operations further comprise:

determining a direction of the second user relative to a computing device that captured the image data; and

adjusting, based on the direction of the second user relative to the computing device that captured the image data, a microphone beamform.

5. The computer-implemented method of claim 4 , wherein adjusting the microphone beamform comprises adjusting the microphone beamform to better isolate and capture the second voice data.

6. The computer-implemented method of claim 1 , wherein the operations further comprise:

determining that the first user is gazing toward a computing device that captured the image data; and

based on determining that the first user is gazing toward the computing device that captured the image data, adjusting a microphone beamform to better isolate and capture the first voice data.

7. The computer-implemented method of claim 1 , wherein the operations further comprise:

determining that the second user is gazing toward a computing device that captured the image data; and

based on determining that the second user is gazing toward the computing device that captured the image data, adjusting a microphone beamform to better isolate and capture the second voice data.

8. The computer-implemented method of claim 1 , wherein the audio data further comprises noises captured from acoustic sources other than the first user and the second user.

9. The computer-implemented method of claim 1 , wherein:

the first annotation comprises a name of the first user; and

the second annotation comprises a name of the second user.

10. The computer-implemented method of claim 1 , wherein:

the first annotation comprises a title of the first user; and

the second annotation comprises a title of the second user.

11. A system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:

obtaining image data comprising a representation of a first user and a second user;

obtaining audio data comprising:

a first voice data corresponding to the first user speaking; and

a second voice data corresponding to the second user speaking;

associating, based on the image data, the first voice data to a first voice-recognition database of the first user speaking and the second voice data to a second voice-recognition database of the second user speaking;

generating, using speech-to-text conversion, a transcription of the audio data;

annotating, based on the first voice data associated with the first voice- recognition database, a first portion of the transcription corresponding to the first voice data with a first annotation identifying the first user; and

annotating, based on the second voice data associated with the second voice-recognition database, a second portion of the transcription corresponding to the second voice data with a second annotation identifying the second user.

12. The system of claim 11 , wherein the operations further comprise:

determining a direction of the first user relative to a computing device that captured the image data; and

adjusting, based on the direction of the first user relative to the computing device that captured the image data, a microphone beamform.

13. The system of claim 12 , wherein adjusting the microphone beamform comprises adjusting the microphone beamform to better isolate and capture the first voice data.

14. The system of claim 11 , wherein the operations further comprise:

determining a direction of the second user relative to a computing device that captured the image data; and

adjusting, based on the direction of the second user relative to the computing device that captured the image data, a microphone beamform.

15. The system of claim 14 , wherein adjusting the microphone beamform comprises adjusting the microphone beamform to better isolate and capture the second voice data.

16. The system of claim 11 , wherein the operations further comprise:

determining that the first user is gazing toward a computing device that captured the image data; and

based on determining that the first user is gazing toward the computing device that captured the image data, adjusting a microphone beamform to better isolate and capture the first voice data.

17. The system of claim 11 , wherein the operations further comprise:

determining that the second user is gazing toward a computing device that captured the image data; and

based on determining that the second user is gazing toward the computing device that captured the image data, adjusting a microphone beamform to better isolate and capture the second voice data.

18. The system of claim 11 , wherein the audio data further comprises noises captured from acoustic sources other than the first user and the second user.

19. The system of claim 11 , wherein:

the first annotation comprises a name of the first user; and

the second annotation comprises a name of the second user.

20. The system of claim 11 , wherein:

the first annotation comprises a title of the first user; and

the second annotation comprises a title of the second user.

Continuity (6)

References Cited (91)