IP Library › Granted Patent US 11,817,084

Granted Patent B2

US 11,817,084 · App. 16/880,647 · Granted Nov 14, 2023

Adaptive interface in a voice-based networked system

Inventors: Pu-sen Chao (Los Altos, CA); Diego Melendo Casado (Mountain View, CA); Ignacio Lopez Moreno (New York, NY)

Assignee: GOOGLE LLC

G10L15/14 G06F3/167 G10L15/005 G10L15/02 G10L15/183 G10L15/1822 G10L15/22 G10L15/30 G10L2015/088 G10L2015/223 G10L2015/228

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 11,817,084
App. No.: 16/880,647
Granted: Nov 14, 2023
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

The present disclosure relates generally to determining a language for speech recognition of a spoken utterance, received via an automated assistant interface, for interacting with an automated assistant. The system can enable multilingual interaction with the automated assistant, without necessitating a user explicitly designate a language to be utilized for each interaction. Selection of a speech recognition model for a particular language can based on one or more interaction characteristics exhibited during a dialog session between a user and an automated assistant. Such interaction characteristics can include anticipated user input types, anticipated user input durations, a duration for monitoring for a user response, and/or an actual duration of a provided user response.

Claims (72)

1. A system to generate voice-based interfaces in networked systems, comprising:

a data processing system having one or more processors coupled with memory to:

receive first audio data corresponding to a first spoken utterance acquired via a sensor of a client device, subsequent to processing second audio data corresponding to a second spoken utterance;

identify an elapsed time between receipt of the first audio data and processing of the second audio data;

determine that the elapsed time is greater than a threshold time;

select, from a plurality of speech recognition models, a first speech recognition model for a first language based on the first audio data, wherein:

the second audio data was processed by a second speech recognition model for a second language; and

the first speech recognition model is selected to be different from the second speech recognition model based on the determination that the elapsed time is greater than the threshold time;

process the first audio data using the first speech recognition model for the first language to identify a first request;

select, from a plurality of digital components, a digital component based on the first request identified from the first audio data; and

provide, to the client device for presentation, the digital component in response to the first audio data.

2. The system of claim 1 , comprising the data processing system to:

determine a score associating the first spoken utterance with the second language, wherein the association is lessened based on the determination that the elapsed time is greater than the threshold time; and

select the first speech recognition model from the plurality of speech recognition models based on the score.

3. The system of claim 2 , wherein, when the first spoken utterance is received, the second speech recognition model is an active model, and wherein the first speech recognition model is selected to be the active model responsive to the score falling below a threshold score.

4. The system of claim 3 , wherein when the score for the active model reaches or falls below a score for an inactive model, the first spoken utterance is processed according to multiple speech recognition models in order to determine a more suitable speech recognition model than the active model.

5. The system of claim 1 , comprising the data processing system to:

identify an interaction characteristic associated with the first audio data corresponding to the first spoken utterance, the interaction characteristic including at least one of a monitoring duration and a response duration; and

identify the first speech recognition model from the plurality of speech recognition model based on a comparison of the interaction characteristic to a reference interaction characteristic.

6. The system of claim 1 , comprising the data processing system to:

process the first audio data using the first speech recognition model for the first language to identify a keyword in the first request, the first keyword in the first language; and

select, from the plurality of digital components, the digital component based on the keyword identified in the first request.

7. The system of claim 1 , comprising the data processing system to:

determine, using the first speech recognition model, a first probability metric that the first spoken utterance is in the first language;

determine, using the second speech recognition model, a second probability metric that the first spoken utterance is in the second language; and

identify the first speech recognition model from the plurality of speech recognition models based on the first probability metric and the second probability metric.

8. The system of claim 1 , comprising the data processing system to select the digital component in the first language corresponding to the first spoken utterance, the digital component different from a second digital component selected in response to the second audio data, the second digital component in the second language corresponding to the second spoken utterance.

9. The system of claim 1 , comprising the data processing system to:

provide, to the client device for presentation, a prompt to receive additional input in response to the second audio data; and

receive, from the client device, the first audio data generated in response to presentation of the prompt on the client device.

10. A method of generating voice-based interfaces in networked systems, comprising:

receiving, by a data processing system, first audio data corresponding to a first spoken utterance acquired via a sensor of a client device, subsequent to processing second audio data corresponding to a second spoken utterance;

identifying, by the data processing system, an elapsed time between receipt of the first audio data and processing of the second audio data;

determining, by the data processing system, that the elapsed time is greater than a threshold time;

selecting, by the data processing system, from a plurality of speech recognition models, a first speech recognition model for a first language based on the first audio data, wherein:

the second audio data was processed by a second speech recognition model for a second language; and

the first speech recognition model is selected to be different from the second speech recognition model based on the determination that the elapsed time is greater than the threshold time;

processing, by the data processing system, the first audio data using the first speech recognition model for the first language to identify a first request;

selecting, by the data processing system, from a plurality of digital components, a digital component based on the first request identified from the first audio data; and

providing, by the data processing system, to the client device for presentation, the digital component in response to the first audio data.

11. The method of claim 10 , comprising:

determining, by the data processing system, a score associating the first spoken utterance with the second language, wherein the association is lessened based on the determination that the elapsed time is greater than the threshold time; and

selecting, by the data processing system, the first speech recognition model from the plurality of speech recognition models based on the score.

12. The method of claim 11 , wherein, when the first spoken utterance is received, the second speech recognition model is an active model, and wherein the first speech recognition model is selected to be the active model responsive to the score falling below a threshold score.

13. The method of claim 10 , comprising:

determining, by the data processing system, that the first spoken utterance corresponds to a first user differing from a second user corresponding to the second spoken utterance based on the first audio data and the second audio data; and

identifying, by the data processing system, the first speech recognition model from the plurality of speech recognition model based on the determination that the first spoken utterance corresponds to the first user differing from the second user.

14. The method of claim 12 , wherein when the score for the active model reaches or falls below a score for an inactive model, the first spoken utterance is processed according to multiple speech recognition models in order to determine a more suitable speech recognition model than the active model.

15. The method of claim 10 , comprising:

processing, by the data processing system, the first audio data using the first speech recognition model for the first language to identify a keyword in the first request, the first keyword in the first language; and

selecting, by the data processing system, from the plurality of digital components, the digital component based on the keyword identified in the first request.

16. The method of claim 10 , comprising

determining, by the data processing system, using the first speech recognition model, a first probability metric that the first spoken utterance is in the first language;

determining, by the data processing system, using the second speech recognition model, a second probability metric that the first spoken utterance is in the second language; and

identifying, by the data processing system, the first speech recognition model from the plurality of speech recognition models based on the first probability metric and the second probability metric.

17. The method of claim 10 , comprising selecting, by the data processing system, the digital component in the first language corresponding to the first spoken utterance, the digital component different from a second digital component selected in response to the second audio data, the second digital component in the second language corresponding to the second spoken utterance.

18. The method of claim 10 , comprising:

providing, by the data processing system, to the client device for presentation, a prompt to receive additional input in response to the second audio data; and

receiving, by the data processing system, from the client device, the first audio data generated in response to presentation of the prompt on the client device.

19. One or more non-transitory computer-readable media storing instructions executable to cause a data processing system to perform operations, the operations comprising:

receiving, by the data processing system, first audio data corresponding to a first spoken utterance acquired via a sensor of a client device, subsequent to processing second audio data corresponding to a second spoken utterance;