Granted Patent B1

US 12,658,191 · App. 18/191,348 · Granted Jun 16, 2026

Inventors: Jing Liu (Pittsburgh, PA); Qi Luo (Seattle, WA); Xinyu Ren (Etobicoke, CA); Ariya Rastrow (Seattle, WA); Ankur Gandhe (Bothell, WA); Denis Filimonov (Seattle, WA); Grant Strimel (Presto, PA); Andreas Stolcke (Berkeley, CA); Ivan Bulyko (Mercer Island, WA); Rahul Pandey (Fairfax, VA)

Assignee: Amazon Technologies, Inc.

G10L17/22G10L17/02G10L17/18

Abstract

Claims (103)

1 . A computer-implemented method comprising:

receiving input audio data representing an utterance;

determining a user profile corresponding to a speaker of the utterance;

determining a first entity associated with the user profile, the first entity corresponding to a first name;

retrieving a first feature vector representing a plurality of graphemes associated with the first name;

retrieving a second feature vector representing a plurality of phonemes associated with pronunciation of the first name;

processing the input audio data using an audio encoder to determine first encoded audio data and second encoded audio data;

processing the first feature vector, the second feature vector, and the first encoded audio data to determine a bias vector;

processing the bias vector and the second encoded audio data to determine third encoded audio data; and

processing the third encoded audio data using a recurrent neural network to determine a transcription of the utterance.

2 . The computer-implemented method of claim 1 , further comprising:

processing the input audio data to determine:

first interim encoded audio data output by a first layer of the audio encoder, and

second interim encoded audio data output by a second layer of the audio encoder;

determining a first plurality of weight values;

determining a second plurality of weight values;

processing the first interim encoded audio data and the first plurality of weight values to determine first weighted interim encoded audio data;

processing the second interim encoded audio data and the second plurality of weight values to determine second weighted interim encoded audio data; and

determining the first encoded audio data using the first weighted interim encoded audio data and the second weighted interim encoded audio data.

3 . The computer-implemented method of claim 2 , further comprising:

using the first interim encoded audio data and the second interim encoded audio data to determine output encoded audio data as output from a final layer of the audio encoder, wherein the second encoded audio data comprises the output encoded audio data.

4 . The computer-implemented method of claim 1 , further comprising:

processing the first encoded audio data using a query component of a biasing component;

determining a third feature vector by concatenating the first feature vector and the second feature vector;

processing the third feature vector using a key component of the biasing component; and

processing the first feature vector using a value component of the biasing component,

wherein the bias vector is determined by the biasing component.

5 . A computer-implemented method comprising:

receiving input audio data representing an utterance;

determining first data corresponding to text representing a first entity;

determining second data corresponding to a pronunciation representing the first entity;

determining third data corresponding to text representing a second entity;

determining fourth data corresponding to a pronunciation representing the second entity;

processing the first data, the second data, the third data, and the fourth data to determine bias data corresponding to a context of the utterance;

processing the input audio data to determine first encoded audio data;

processing the first encoded audio data and the bias data to determine adjusted first encoded audio data; and

processing the adjusted first encoded audio data to determine output data representing a transcription of the utterance, wherein the output data represents the first entity.

6 . The computer-implemented method of claim 5 , further comprising:

determining a user profile associated with the input audio data;

determining the user profile is associated with the first entity and the second entity; and

based at least in part on the user profile being associated with the first entity and the second entity, selecting the first data, the second data, the third data, and the fourth data for processing to determine the bias data.

7 . The computer-implemented method of claim 5 , further comprising:

determining a display associated with an utterance is presenting information related to the first entity and the second entity; and

based at least in part on the display is presenting information related to the first entity and the second entity, selecting the first data, the second data, the third data, and the fourth data for processing to determine the bias data.

8 . The computer-implemented method of claim 5 , further comprising:

processing the input audio data to determine:

first interim encoded audio data output by a first layer of an audio encoder, and

second interim encoded audio data output by a second layer of the audio encoder;

determining a first plurality of weight values;

determining a second plurality of weight values;

processing the first interim encoded audio data and the first plurality of weight values to determine first weighted interim encoded audio data;

processing the second interim encoded audio data and the second plurality of weight values to determine second weighted interim encoded audio data; and

determining second encoded audio data using the first weighted interim encoded audio data and the second weighted interim encoded audio data,

wherein the second encoded audio data is also processed to determine the bias data.

9 . The computer-implemented method of claim 8 , further comprising:

using the first interim encoded audio data and the second interim encoded audio data to determine output encoded audio data as output from a final layer of the audio encoder, wherein the first encoded audio data comprises the output encoded audio data.

10 . The computer-implemented method of claim 8 , further comprising:

processing the second encoded audio data using a query component of a biasing component;

processing the first data and the second data using a key component of the biasing component; and

processing the first data using a value component of the biasing component,

wherein the bias data is determined by the biasing component.

11 . The computer-implemented method of claim 5 , further comprising:

processing the adjusted first encoded audio data using a recurrent neural network to determine output distribution data and output probability data; and

processing the output distribution data and output probability data using a selection component to determine the output data.

12 . A system comprising:

at least one processor; and

memory including instructions operable to be executed by the at least one processor to cause the system to:

receive input audio data representing an utterance;

determine first data corresponding to a text representing a first entity;

determine second data corresponding to a pronunciation representing the first entity;

determine third data corresponding to text representing a second entity;

determine fourth data corresponding to a pronunciation representing the second entity;

process the first data, the second data, the third data, and the fourth data to determine bias data corresponding to a context of the utterance;

process the input audio data to determine first encoded audio data;

process the first encoded audio data and the bias data to determine adjusted first encoded audio data; and

process the adjusted first encoded audio data to determine output data representing a transcription of the utterance, wherein the output data represents the first entity.

13 . The system of claim 12 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine a user profile associated with the input audio data;

determine the user profile is associated with the first entity and the second entity; and

based at least in part on the user profile being associated with the first entity and the second entity, select the first data, the second data, the third data, and the fourth data for processing to determine the bias data.

14 . The system of claim 12 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine a display associated with an utterance is presenting information related to the first entity and the second entity; and

based at least in part on the display is presenting information related to the first entity and the second entity, select the first data, the second data, the third data, and the fourth data for processing to determine the bias data.

15 . The system of claim 12 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

process the input audio data to determine:

first interim encoded audio data output by a first layer of an audio encoder, and

second interim encoded audio data output by a second layer of the audio encoder;

determine a first plurality of weight values;

determine a second plurality of weight values;

process the first interim encoded audio data and the first plurality of weight values to determine first weighted interim encoded audio data;

process the second interim encoded audio data and the second plurality of weight values to determine second weighted interim encoded audio data; and

determine second encoded audio data using the first weighted interim encoded audio data and the second weighted interim encoded audio data,

wherein the second encoded audio data is also processed to determine the bias data.

16 . The system of claim 15 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

use the first interim encoded audio data and the second interim encoded audio data to determine output encoded audio data as output from a final layer of the audio encoder, wherein the first encoded audio data comprises the output encoded audio data.

17 . The system of claim 15 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

process the second encoded audio data using a query component of a biasing component;

process the first data and the second data using a key component of the biasing component; and

process the first data using a value component of the biasing component,

wherein the bias data is determined by the biasing component.

18 . The system of claim 12 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

process the adjusted first encoded audio data using a recurrent neural network to determine output distribution data and output probability data; and

process the output distribution data and output probability data using a selection component to determine the output data.

Assignments (1)

Continuity (1)

References Cited (63)