Granted Patent B2

US 12,283,278 · App. 18/615,621 · Granted Apr 22, 2025

Inventors: Benjamin Haynor (New York, NY); Petar Aleksic (Jersey City, NJ)

Assignee: GOOGLE LLC

G10L15/26G10L15/16G10L15/193G10L15/22G10L15/30

Abstract

Claims (64)

1. A client device comprising:

one or more microphones;

memory;

one or more processors that execute instructions, stored in the memory, to:

generate a text representation of audio data capturing a spoken utterance, including an alphanumeric sequence, using an automatic speech recognition (“ASR”) engine, wherein in generating the text representation of the audio data capturing the spoken utterance, including the alphanumeric sequence, using the ASR engine, comprises:

determine contextual information for the alphanumeric sequence, wherein in determining the contextual information for the alphanumeric sequence is based on a rendered system prompt that immediately preceded the spoken utterance, and wherein determining the contextual information based on the rendered system prompt one or more of the processors are to:

determine the contextual information based on at least one predicted response to the rendered system prompt;

select, based on the contextual information, one or more contextual finite state transducers for the alphanumeric sequence;

generate a set of candidate recognitions of the spoken utterance based on processing the audio data using an ASR model portion of the ASR engine; and

generate the text representation of the spoken utterance, wherein the text representation includes the alphanumeric sequence, and wherein generating the text representation is based on the generated set of candidate recognitions and the one or more selected contextual finite state transducers.

2. The client device of claim 1 , wherein the ASR model is a recurrent neural network transducer (RNN-T) model, and wherein the ASR engine further comprises a beam search portion.

3. The client device of claim 2 , wherein in generating the text representation of the spoken utterance based on the generated set of candidate recognitions and the one or more selected contextual finite state transducers one or more of the processors are to:

modify the beam search portion of the ASR engine using the one or more contextual finite state transducers.

4. The client device of claim 3 , wherein in generating the text representation of the spoken utterance based on the generated set of candidate recognitions and the one or more selected contextual finite state transducers one or more of the processors are further to:

determine a corresponding probability measure for each candidate recognition in the set of candidate recognitions;

modify the corresponding probability measures using the beam search portion of the ASR engine modified using the one or more contextual finite state transducers;

select a candidate recognition, from the set of candidate recognitions, based on determining that the corresponding probability measure for the candidate recognition satisfies one or more conditions; and

generate the text representation of the spoken utterance based on the selected candidate recognition.

5. The client device of claim 1 , wherein the audio data is captured via the one or more microphones of the client device.

6. The client device of claim 1 , wherein determining the contextual information for the alphanumeric sequence is based on the audio data capturing the spoken utterance, and wherein in determining the contextual information for the alphanumeric sequence based on the audio data one or more of the processors are to:

generate the contextual information for the alphanumeric sequence based on one or more recognized terms of the audio data, the one or more recognized terms being in addition to the alphanumeric sequence.

7. The client device of claim 1 , wherein one or more of the processors are further to generate at least a given contextual finite state transducer, of the one or more contextual finite state transducers, wherein in generating the given contextual finite state transducer one or more of the processors are to:

select an alphanumeric grammar finite state transducer corresponding to an alphanumeric sequence;

select a speller finite state transducer which maps wordpieces to constitute graphemes; and

generate an unweighted wordpiece based acceptor grammar based on the alphanumeric grammar finite state transducer and the speller finite state transducer.

8. The client device of claim 1 , wherein the alphanumeric sequence includes at least one number and includes at least one letter.

9. The client device of claim 1 , wherein the ASR model portion of the ASR engine is an end-to-end speech recognition model.

10. The client device of claim 1 , wherein the ASR engine is trained using a set of training instances, and wherein

the alphanumeric sequence is not in the set of training instances; or

the alphanumeric sequence occurs a number of times, in the set of training instances, that is below a threshold value.

11. A client device comprising:

one or more microphones;

memory;

one or more processors that execute instructions, stored in the memory, to:

generate a text representation of audio data capturing a spoken utterance, including an alphanumeric sequence, using an automatic speech recognition (“ASR”) engine, wherein in generating the text representation of the audio data capturing the spoken utterance, including the alphanumeric sequence, using the ASR engine, comprises:

determine contextual information for the alphanumeric sequence, wherein in determining the contextual information for the alphanumeric sequence is based on a rendered system prompt that immediately preceded the spoken utterance, and wherein determining the contextual information based on the rendered system prompt one or more of the processors are to:

determine the contextual information based on at least one predicted response to the rendered system prompt;

select, based on the contextual information, one or more contextual finite state transducers for the alphanumeric sequence;

generate a set of candidate recognitions of the spoken utterance based on processing the audio data using an ASR model portion of the ASR engine;

and generate the text representation of the spoken utterance, wherein the text representation includes the alphanumeric sequence, and wherein generating the text representation is based on the generated set of candidate recognitions and the one or more selected contextual finite state transducers.

12. The system of claim 11 , wherein the ASR model is a recurrent neural network transducer (RNN-T) model, and wherein the ASR engine further comprises a beam search portion.

13. The system of claim 12 , wherein in generating the text representation of the spoken utterance based on the generated set of candidate recognitions and the one or more selected contextual finite state transducers one or more of the processors are to:

modify the beam search portion of the ASR engine using the one or more contextual finite state transducers.

14. The system of claim 13 , wherein in generating the text representation of the spoken utterance based on the generated set of candidate recognitions and the one or more selected contextual finite state transducers one or more of the processors are further to:

determine a corresponding probability measure for each candidate recognition in the set of candidate recognitions;

modify the corresponding probability measures using the beam search portion of the ASR engine modified using the one or more contextual finite state transducers;

select a candidate recognition, from the set of candidate recognitions, based on determining that the corresponding probability measure for the candidate recognition satisfies one or more conditions; and

generate the text representation of the spoken utterance based on the selected candidate recognition.

15. The system of claim 11 , wherein the audio data is captured via the one or more microphones of the client device.

16. The system of claim 11 , wherein determining the contextual information for the alphanumeric sequence is based on the audio data capturing the spoken utterance, and wherein in determining the contextual information for the alphanumeric sequence based on the audio data one or more of the processors are to:

generate the contextual information for the alphanumeric sequence based on one or more recognized terms of the audio data, the one or more recognized terms being in addition to the alphanumeric sequence.

17. The system of claim 11 , wherein in determining the contextual information for the alphanumeric sequence is based on a rendered system prompt that immediately preceded the spoken utterance, and wherein determining the contextual information based on the rendered system prompt one or more of the processors are to:

determine the contextual information based on at least one predicted response to the rendered system prompt.

18. The system of claim 11 , wherein one or more of the processors are further to generate at least a given contextual finite state transducer, of the one or more contextual finite state transducers, wherein in generating the given contextual finite state transducer one or more of the processors are to:

select an alphanumeric grammar finite state transducer corresponding to an alphanumeric sequence;

select a speller finite state transducer which maps wordpieces to constitute graphemes; and

generate an unweighted wordpiece based acceptor grammar based on the alphanumeric grammar finite state transducer and the speller finite state transducer.

19. A method implemented by one or more processors, the method comprising:

generating a text representation of audio data capturing a spoken utterance, including an alphanumeric sequence, using an automatic speech recognition (“ASR”) engine, wherein in generating the text representation of the audio data capturing the spoken utterance, including the alphanumeric sequence, using the ASR engine, comprises:

determining contextual information for the alphanumeric sequence, wherein in determining the contextual information for the alphanumeric sequence is based on a rendered system prompt that immediately preceded the spoken utterance, and wherein determining the contextual information based on the rendered system prompt one or more of the processors are to:

determine the contextual information based on at least one predicted response to the rendered system prompt;

selecting, based on the contextual information, one or more contextual finite state transducers for the alphanumeric sequence;

generating a set of candidate recognitions of the spoken utterance based on processing the audio data using an ASR model portion of the ASR engine; and

generating the text representation of the spoken utterance, wherein the text representation includes the alphanumeric sequence, and wherein generating the text representation is based on the generated set of candidate recognitions and the one or more selected contextual finite state transducers.

Assignments (1)

Continuity (2)

References Cited (33)