Interactive decoding of words from phoneme score distributions
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for interactive decoding of a word sequence.
1 . A method for performing automatic speech recognition and performed by one or more computers, the method comprising:
processing an input comprising one or more of a video of a speaker speaking an utterance or audio of the utterance using a speech recognition neural network to generate, for each time step in a sequence of time steps of the utterance, a respective phoneme score distribution for the time step that assigns a respective score to each of a plurality of phoneme tokens, the phoneme tokens comprising (i) a plurality of phonemes and (ii) a blank symbol that indicates that no phoneme is spoken at the time step;
generating, using the respective phoneme score distributions, a word sequence of words that represents a decoded word sequence of the utterance, the generating comprising:
initializing fringe data that specifies a plurality of states, wherein each state identifies (i) a candidate sequence of phonemes and (ii) a corresponding candidate sequence of words that is represented by the phonemes; and
generating the word sequence by adding a new word to the word sequence at each of a plurality of update iterations, wherein performing a particular update iteration comprises:
updating the fringe data using the respective phoneme score distributions by, for each state in the fringe data, extending, using the respective phoneme score distributions that are generated by the speech recognition neural network and that each assign a respective score to each of the plurality of phoneme tokens, the candidate sequence of words identified by the state to include a respective additional candidate word after the last word in the word sequence as of the particular update iteration;
providing, for presentation on a user device, one or more of the additional candidate words specified by the states specified in the fringe data;
receiving, from the user device, a user selection of one of the additional candidate words;
responsive to the user selection of the additional candidate word:
updating the word sequence by adding the additional candidate word selected by the user after the last word of the word sequence; and
removing, from the fringe data, any state that does not identify a candidate sequence of words that ends in the additional candidate word selected by the user.
2 . The method of claim 1 , wherein obtaining, for each time step in a sequence of time steps, a respective phoneme score distribution comprises:
processing a video of a speaker using a visual speech recognition neural network that is configured to process the video to generate the respective phoneme score distributions.
3 . The method of claim 1 , wherein obtaining, for each time step in a sequence of time steps, a respective phoneme score distribution comprises:
processing an audio input representing an utterance using an audio speech recognition neural network that is configured to process the audio input to generate the respective phoneme score distributions.
4 . The method of claim 1 , wherein generating the word sequence by adding a new word to the word sequence at each of a plurality of update iterations comprises performing update iterations until a user input is received.
5 . The method of claim 1 , wherein generating the word sequence by adding a new word to the word sequence at each of a plurality of update iterations comprises performing update iterations until the candidate sequences of phonemes identified by all of the states in the fringe data have been generated using all of the respective phoneme distributions at all of the time steps.
6 . The method of claim 1 , wherein updating the fringe data using the respective phoneme score distributions comprises:
after removing from the fringe data any states that do not identify a candidate sequence of words that ends in the last word in the word sequence as of the particular update iteration, repeatedly performing operations until each state specified by the fringe data identifies an additional candidate word after the last word in the word sequence as of the particular update iteration, the operations comprising:
for each particular state in the fringe data that does not yet end in an additional candidate word after the last word in the word sequence as of the particular update iteration:
generating, from the particular state, a plurality of new candidate states; and
generating a respective ranking score for each new candidate state using the respective score distributions; and
updating the fringe data to only specify a predetermined number of states with the highest ranking scores.
7 . The method of claim 6 , wherein providing, for presentation on a user device, the additional candidate words specified by the states specified in the fringe data comprises:
providing the additional candidate words for presentation in an order according to the ranking scores for the corresponding states.
8 . The method of claim 6 , wherein generating, from the particular state, a plurality of new candidate states comprises, for each phoneme token:
generating a new state that identifies a candidate phoneme sequence that includes the phonemes in the candidate sequence identified by the particular state followed by the phoneme token.
9 . The method of claim 8 , wherein each state further identifies (iii) a time step in the sequence of time steps, and wherein generating a respective ranking score for each new candidate state using the respective probability distribution comprises:
generating the respective ranking score for each new state generated from a given existing state from the respective score assigned to the phoneme token that was added to the given existing state at the time step immediately following the time step identified by the given existing state, wherein the new state identifies the time step immediately following the time step identified by the existing state.
10 . The method of claim 9 , wherein the respective ranking score for each new state is further based on a finite state transducer (FST) path weight assigned to the new state by a decoder FST based on a relation between the corresponding candidate phoneme sequence for the new state and the corresponding candidate word sequence for the new state.
11 . One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
processing an input comprising one or more of a video of a speaker speaking an utterance or audio of the utterance using a speech recognition neural network to generate, for each time step in a sequence of time steps of the utterance, a respective phoneme score distribution for the time step that assigns a respective score to each of a plurality of phoneme tokens, the phoneme tokens comprising (i) a plurality of phonemes and (ii) a blank symbol that indicates that no phoneme is spoken at the time step;
generating, using the respective phoneme score distributions, a word sequence of words that represents a decoded word sequence of the utterance, the generating comprising:
initializing fringe data that specifies a plurality of states, wherein each state identifies (i) a candidate sequence of phonemes and (ii) a corresponding candidate sequence of words that is represented by the phonemes; and
generating the word sequence by adding a new word to the word sequence at each of a plurality of update iterations, wherein performing a particular update iteration comprises:
updating the fringe data using the respective phoneme score distributions by, for each state in the fringe data, extending, using the respective phoneme score distributions that are generated by the speech recognition neural network and that each assign a respective score to each of the plurality of phoneme tokens, the candidate sequence of words identified by the state to include a respective additional candidate word after the last word in the word sequence as of the particular update iteration;
providing, for presentation on a user device, one or more of the additional candidate words specified by the states specified in the fringe data;
receiving, from the user device, a user selection of one of the additional candidate words;
responsive to the user selection of the additional candidate word:
updating the word sequence by adding the additional candidate word selected by the user after the last word of the word sequence; and
removing, from the fringe data, any state that does not identify a candidate sequence of words that ends in the additional candidate word selected by the user.
12 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
processing an input comprising one or more of a video of a speaker speaking an utterance or audio of the utterance using a speech recognition neural network to generate, for each time step in a sequence of time steps of the utterance, a respective phoneme score distribution for the time step that assigns a respective score to each of a plurality of phoneme tokens, the phoneme tokens comprising (i) a plurality of phonemes and (ii) a blank symbol that indicates that no phoneme is spoken at the time step;
generating, using the respective phoneme score distributions, a word sequence of words that represents a decoded word sequence of the utterance, the generating comprising:
initializing fringe data that specifies a plurality of states, wherein each state identifies (i) a candidate sequence of phonemes and (ii) a corresponding candidate sequence of words that is represented by the phonemes; and
generating the word sequence by adding a new word to the word sequence at each of a plurality of update iterations, wherein performing a particular update iteration comprises:
updating the fringe data using the respective phoneme score distributions by, for each state in the fringe data, extending, using the respective phoneme score distributions that are generated by the speech recognition neural network and that each assign a respective score to each of the plurality of phoneme tokens, the candidate sequence of words identified by the state to include a respective additional candidate word after the last word in the word sequence as of the particular update iteration;
providing, for presentation on a user device, one or more of the additional candidate words specified by the states specified in the fringe data;
receiving, from the user device, a user selection of one of the additional candidate words;
responsive to the user selection of the additional candidate word:
updating the word sequence by adding the additional candidate word selected by the user after the last word of the word sequence; and
removing, from the fringe data, any state that does not identify a candidate sequence of words that ends in the additional candidate word selected by the user.
13 . The system of claim 12 , wherein obtaining, for each time step in a sequence of time steps, a respective phoneme score distribution comprises:
processing a video of a speaker using a visual speech recognition neural network that is configured to process the video to generate the respective phoneme score distributions.
14 . The system of claim 12 , wherein obtaining, for each time step in a sequence of time steps, a respective phoneme score distribution comprises:
processing an audio input representing an utterance using an audio speech recognition neural network that is configured to process the audio input to generate the respective phoneme score distributions.
15 . The system of claim 12 , wherein generating the word sequence by adding a new word to the word sequence at each of a plurality of update iterations comprises performing update iterations until a user input is received.
16 . The system of claim 12 , wherein generating the word sequence by adding a new word to the word sequence at each of a plurality of update iterations comprises performing update iterations until the candidate sequences of phonemes identified by all of the states in the fringe data have been generated using all of the respective phoneme distributions at all of the time steps.
17 . The system of claim 12 , wherein updating the fringe data using the respective phoneme score distributions comprises:
after removing from the fringe data any states that do not identify a candidate sequence of words that ends in the last word in the word sequence as of the particular update iteration, repeatedly performing operations until each state specified by the fringe data identifies an additional candidate word after the last word in the word sequence as of the particular update iteration, the operations comprising:
for each particular state in the fringe data that does not yet end in an additional candidate word after the last word in the word sequence as of the particular update iteration:
generating, from the particular state, a plurality of new candidate states; and
generating a respective ranking score for each new candidate state using the respective score distributions; and
updating the fringe data to only specify a predetermined number of states with the highest ranking scores.
18 . The system of claim 17 , wherein providing, for presentation on a user device, the additional candidate words specified by the states specified in the fringe data comprises:
providing the additional candidate words for presentation in an order according to the ranking scores for the corresponding states.