Guided data selection for masked speech modeling based on an average score assigned to encoded representations of an utterance
A method of guided data selection for masked speech modeling includes obtaining a sequence of encoded representations corresponding to an utterance. For each respective encoded representation, the method includes processing the respective encoded representation to generate a corresponding probability distribution over possible speech recognition hypotheses and assigning, to the respective encode representation, a confidence score as a highest probability from the corresponding probability distribution over possible speech recognition hypotheses. The method also includes selecting a set of unmasked encoded representations to mask based on the confidence scores assigned to the sequence of encoded representations. The method also includes generating a set of masked encoded representations by masking the selected set of unmasked encoded representations. Here, each masked encoded representation in the set of masked encoded representations corresponds to a respective one of the unmasked encoded representations in the selected set of unmasked encoded representations.
1 . A computer-implemented method for guided data selection for masked speech modeling, the computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:
obtaining a sequence of encoded representations corresponding to an utterance;
for each respective encoded representation in the sequence of encoded representations:
processing, using a scorer model, the respective encoded representation to generate a corresponding probability distribution over possible speech recognition hypotheses for the respective encoded representation; and
assigning, to the respective encoded representation, a confidence score as a highest probability from the corresponding probability distribution over possible speech recognition hypotheses for the respective encoded representation;
based on the confidence scores assigned to the sequence of encoded representations, selecting, from the sequence of encoded representations, a set of unmasked encoded representations to mask;
generating a set of masked encoded representations by masking the selected set of unmasked encoded representations, wherein each masked encoded representation in the set of masked encoded representations corresponds to a respective one of the unmasked encoded representations in the selected set of unmasked encoded representations;
pre-training an audio encoder based on the set of masked encoded representations,
wherein a speech recognition model comprising the pre-trained audio encoder is configured to generate a speech recognition result for a spoken utterance; and
determining an utterance-level confidence score by averaging the confidence scores assigned to the set of masked encoded representations.
2 . The computer-implemented method of claim 1 , wherein the operations further comprise:
for each respective unmasked encoded representation in the selected set of unmasked encoded representations to mask, generating, using a quantizer, a corresponding target context vector for the respective unmasked encoded representation; and
for each respective masked encoded representation in the set of masked encoded representations:
generating a corresponding contrastive context vector for the respective masked encoded representation; and
generating a contrastive loss based on the corresponding contrastive context vector and the corresponding target context vector generated for the respective unmasked encoded representation that corresponds to the respective masked encoded representation,
wherein pre-training the audio encoder is further based on using the contrastive losses generated for the set of masked encoded representations.
3 . The computer-implemented method of claim 2 , wherein the operations further comprise:
for each respective unmasked encoded representation in the selected set of unmasked encoded representations to mask, generating, using a cluster module, a corresponding K-means cluster for the respective unmasked encoded representation; and
for each respective masked encoded representation in the set of masked encoded representations, generating a cross-entropy loss based on the corresponding contrastive context vector and the corresponding K-means cluster generated for the respective unmasked encoded representation that corresponds to the respective masked encoded representation, and
wherein pre-training the audio encoder is further based on using the cross-entropy losses generated for the set of masked encoded representations.
4 . The computer-implemented method of claim 3 , wherein the operations further comprise:
for each respective masked encoded representation in the set of masked encoded representations, determining a final training objective based on the contrastive loss and the cross-entropy loss,
wherein pre-training the audio encoder is further based on the final training objectives generated for the set of masked encoded representations.
5 . The computer-implemented method of claim 4 , wherein the operations further comprise:
determining an utterance-level confidence score by averaging the confidence scores in the set of masked encoded representations; and
weighting the final training objective based on the utterance-level confidence score,
wherein pre-training the audio encoder is further based on the weighted final training objective.
6 . The computer-implemented method of claim 3 , wherein the operations further comprise extracting bottleneck features from the contrastive context vectors.
7 . The computer-implemented method of claim 6 , wherein the operations further comprise refining each corresponding K-means cluster using the extracted bottleneck features.
8 . The computer-implemented method of claim 1 , wherein selecting the set of unmasked encoded representations to mask comprises selecting the top-K encoded representations from the sequence of encoded representations having the highest confidence scores.
9 . The computer-implemented method of claim 8 , wherein K is based on a predetermined ratio of encoded representations in the sequence of encoded representations to be masked.
10 . The computer-implemented method of claim 9 , wherein the predetermined ratio is equal to forty percent.
11 . A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
obtaining a sequence of encoded representations corresponding to an utterance;
for each respective encoded representation in the sequence of encoded representations:
processing, using a scorer model, the respective encoded representation to generate a corresponding probability distribution over possible speech recognition hypotheses for the respective encoded representation; and
assigning, to the respective encoded representation, a confidence score as a highest probability from the corresponding probability distribution over possible speech recognition hypotheses for the respective encoded representation;
based on the confidence scores assigned to the sequence of encoded representations, selecting, from the sequence of encoded representations, a set of unmasked encoded representations to mask;
generating a set of masked encoded representations by masking the selected set of unmasked encoded representations, wherein each masked encoded representation in the set of masked encoded representations corresponds to a respective one of the unmasked encoded representations in the selected set of unmasked encoded representations;
pre-training an audio encoder based on the set of masked encoded representations,
wherein a speech recognition model comprising the pre-trained audio encoder is configured to generate a speech recognition result for a spoken utterance; and
determining an utterance-level confidence score by averaging the confidence scores assigned to the set of masked encoded representations.
12 . The system of claim 11 , wherein the operations further comprise:
for each respective unmasked encoded representation in the selected set of unmasked encoded representations to mask, generating, using a quantizer, a corresponding target context vector for the respective unmasked encoded representation; and
for each respective masked encoded representation in the set of masked encoded representations:
generating a corresponding contrastive context vector for the respective masked encoded representation; and
generating a contrastive loss based on the corresponding contrastive context vector and the corresponding target context vector generated for the respective unmasked encoded representation that corresponds to the respective masked encoded representation,
wherein pre-training the audio encoder is further based on using the contrastive losses generated for the set of masked encoded representations.
13 . The system of claim 12 , wherein the operations further comprise:
for each respective unmasked encoded representation in the selected set of unmasked encoded representations to mask, generating, using a cluster module, a corresponding K-means cluster for the respective unmasked encoded representation; and
for each respective masked encoded representation in the set of masked encoded representations, generating a cross-entropy loss based on the corresponding contrastive context vector and the corresponding K-means cluster generated for the respective unmasked encoded representation that corresponds to the respective masked encoded representation, and
wherein pre-training the audio encoder is further based on using the cross-entropy losses generated for the set of masked encoded representations.
14 . The system of claim 13 , wherein the operations further comprise:
for each respective masked encoded representation in the set of masked encoded representations, determining a final training objective based on the contrastive loss and the cross-entropy loss,
wherein pre-training the audio encoder is further based on the final training objectives generated for the set of masked encoded representations.
15 . The system of claim 14 , wherein the operations further comprise:
determining an utterance-level confidence score by averaging the confidence scores in the set of masked encoded representations; and
weighting the final training objective based on the utterance-level confidence score,
wherein pre-training the audio encoder is further based on the weighted final training objective.
16 . The system of claim 13 , wherein the operations further comprise extracting bottleneck features from the contrastive context vectors.
17 . The system of claim 16 , wherein the operations further comprise refining each corresponding K-means cluster using the extracted bottleneck features.
18 . The system of claim 11 , wherein selecting the set of unmasked encoded representations to mask comprises selecting the top-K encoded representations from the sequence of encoded representations having the highest confidence scores.
19 . The system of claim 18 , wherein K is based on a predetermined ratio of encoded representations in the sequence of encoded representations to be masked.
20 . The system of claim 19 , wherein the predetermined ratio is equal to forty percent.