Granted Patent B2

US 12,260,852 · App. 18/078,460 · Granted Mar 25, 2025

Inventors: Zengwei Yao (Beijing, CN); Liyong Guo (Beijing, CN); Povey Daniel (Beijing, CN); Long Lin (Beijing, CN); Fangjun Kuang (Beijing, CN); Wei Kang (Beijing, CN); Mingshuang Luo (Beijing, CN); Quandong Wang (Beijing, CN); Yuxiang Kong (Beijing, CN)

Assignee: BEIJING XIAOMI MOBILE SOFTWARE CO., LTD.

G10L15/063G10L15/16G10L15/22G10L19/032G10L2019/0002

Abstract

Claims (115)

1. A method of training a speech recognition model, comprising:

for each of a plurality of training samples,

inputting speech data of a training sample into a teacher model and a to-be-trained speech recognition model separately,

obtaining an embedding outputted by the teacher model and encoded data outputted by the to-be-trained speech recognition model, wherein the embedding comprises a floating-point vector holding D floating-point numbers,

obtaining quantized codebook data by performing a multi-codebook quantization on the embedding, wherein the quantized codebook data comprises N integers corresponding to the speech data and each integer indicates a codebook index, wherein N is a positive integer,

calculating a loss based on the encoded data, the quantized codebook data, and text data in the training sample, and

obtaining a trained speech recognition model by stopping training the to-be-trained speech recognition model in response to determining at least one of followings: the loss being less than or equal to a preset loss threshold, or trained times being greater than preset trained times.

2. The method according to claim 1 , wherein obtaining the quantized codebook data by performing the multi-codebook quantization on the embedding comprises:

converting the embedding into first intermediate data, wherein the first intermediate data comprises N*C integers, wherein N indicates a number of codebooks, and C indicates a number of categories in each codebook;

obtaining initial codebook indexes comprising N integers as second intermediate data by taking a maximum value of each category in N codebooks of the first intermediate data; and

obtaining the quantized codebook data corresponding to a smallest relative reconstruction loss (RRL) by iterating over the second intermediate data.

3. The method according to claim 2 , wherein obtaining the quantized codebook data corresponding to the smallest RRL by iterating over the second intermediate data comprises an i-th iteration process, wherein i is an integer from 1 to K, K indicates a total iteration number, and 2 K is equal to N;

wherein in response to determining that i is equal to 1, a first iteration process comprises:

combining the N codebooks of the first intermediate data in pairs to obtain N/2 1 Level-1 codebook combinations, wherein each Level-1 codebook combination comprises C*C Level-1 element combinations, and each Level-1 element combination comprises 2 1 members;

for each Level-1 codebook combination, obtaining an RRL of each Level-1 element combination and the initial codebook indexes of remaining codebooks in the second intermediate data; and

for each Level-1 codebook combination, taking L Level-1 element combinations corresponding to smallest L RRLs as elements of the Level-1 codebook combination;

wherein in response to determining that i is an integer between 1 and K, the i-th iteration process comprises:

combining Level-(i−1) codebook combinations in pairs to obtain N/2 i Level-i codebook combinations, wherein each Level-i codebook combination comprises L*L Level-i element combinations, and each Level-i element combination comprises 2 i members;

for each Level-i codebook combination, obtaining an RRL of each Level-i element combination and the initial codebook indexes of the remaining codebooks in the second intermediate data; and

for each Level-i codebook combination, taking L Level-i element combinations corresponding to smallest L RRLs as elements of the Level-i codebook combination; and

wherein in response to determining that i is equal to K, a K-th iteration process comprises:

combining Level-(K−1) codebook combinations in pairs to obtain a Level-K codebook combination, wherein the Level-K codebook combination comprises L*L Level-K element combinations, and each Level-K element combination comprises 2 K members;

obtaining an RRL of each Level-K element combination in the Level-K codebook combination; and

taking the Level-K element combination with a smallest RRL as the quantized codebook data.

4. The method according to claim 1 , further comprising:

inputting the embedding into a codebook quantization model that is trained in advance, and

obtaining the quantized codebook data corresponding to a smallest relative reconstruction loss (RRL) outputted by the codebook quantization model after performing the multi-codebook quantization on the embedding.

5. The method according to claim 4 , wherein the codebook quantization model is trained by:

obtaining the quantized codebook data and a first intermediate data outputted by the codebook quantization model by inputting an embedding sample into the codebook quantization model;

obtaining a reconstructed embedding outputted by a quantized-data reconstruction model by inputting the quantized codebook data into the quantized-data reconstruction model;

calculating an RRL based on the embedding sample and the reconstructed embedding, a uniform distribution loss based on the first intermediate data, and a consistency loss based on the first intermediate data and the quantized codebook data; and

completing training the codebook quantization model in response to determining at least one of followings: each of the RRL, the uniform distribution loss and the consistency loss meeting a requirement, or iteration times meeting a preset times threshold.

6. The method according to claim 5 , wherein calculating the RRL based on the embedding sample and the reconstructed embedding comprises:

obtaining first square values of differences between the embedding sample and the reconstructed embedding;

obtaining a sum of the first square values;

obtaining second square values of differences between the embedding sample and a mean value of the embedding sample;

obtaining a sum of the second square values; and

obtaining the RRL by obtaining a ratio of the sum of the first square values to the sum of the second square values.

7. The method according to claim 5 , wherein calculating the uniform distribution loss based on the first intermediate data comprises:

obtaining a product of a predicted distribution probability of each category in each codebook and a logarithm of the predicted distribution probability;

obtaining a first sum of products corresponding to all categories in each codebook;

obtaining a second sum of first sums corresponding to all codebooks; and

obtaining the uniform distribution loss by calculating a mean value of the second sum.

8. The method according to claim 5 , wherein calculating the consistency loss based on the first intermediate data and the quantized codebook data comprises:

obtaining a predicted probability of each category in each iterated codebook;

obtaining a logarithm of the predicted probability;

for logarithms corresponding to all codebooks, obtaining a sum of the logarithms and an inverse of a mean value of the logarithms; and

taking the inverse of the mean value as the consistency loss.

9. The method according to claim 1 , wherein the loss comprises a categorical cross entropy, and calculating the loss based on the encoded data, the quantized codebook data and the text data in the training sample comprises:

obtaining a probability of each category in each codebook of the encoded data after a conversion;

obtaining a logarithm of each probability in each codebook;

calculating a sum of logarithms of all codebooks;

calculating a mean value of the sum; and

taking an inverse of the mean value as the categorical cross entropy.

10. The method according to claim 1 , wherein the loss comprises a Recurrent Neural Network Transducer (RNN-T) loss function value, and calculating the loss based on the encoded data, the quantized codebook data and the text data in the training sample comprises:

obtaining a probability of each alignment scheme as enumerating alignment schemes between the speech data and the text data; and

obtaining a sum of probabilities corresponding to all the enumerating alignment schemes as the RNN-T loss function value.

11. A speech recognition method, comprising:

acquiring speech data;

inputting the speech data into a preset speech recognition model; and

obtaining text data outputted by the speech recognition model;

wherein the speech recognition model is trained and implemented by adopting the training method according to claim 1 .

12. An electronic device, comprising:

a memory and one or more processors;

wherein the memory is configured to store a computer program executable by the one or more processors; and

wherein the one or more processors are configured to execute the computer program in the memory to implement acts comprising:

for each of a plurality of training samples,

inputting speech data of a training sample into a teacher model and a to-be-trained speech recognition model separately,

obtaining an embedding outputted by the teacher model and encoded data outputted by the to-be-trained speech recognition model, wherein the embedding comprises a floating-point vector holding D floating-point numbers,

obtaining quantized codebook data by performing a multi-codebook quantization on the embedding, wherein the quantized codebook data comprises N integers corresponding to the speech data and each integer indicates a codebook index, wherein N is a positive integer,

calculating a loss based on the encoded data, the quantized codebook data, and text data in the training sample, and

obtaining a trained speech recognition model by stopping training the to-be-trained speech recognition model in response to determining at least one of followings: the loss being less than or equal to a preset loss threshold, or trained times being greater than preset trained times.

13. The electronic device according to claim 12 , wherein the one or more processors, when obtaining the quantized codebook data by performing the multi-codebook quantization on the embedding, are further configured to implement:

converting the embedding into first intermediate data, wherein the first intermediate data comprises N*C integers, wherein N indicates a number of codebooks, and C indicates a number of categories in each codebook;

obtaining initial codebook indexes comprising N integers as second intermediate data by taking a maximum value of each category in N codebooks of the first intermediate data; and

obtaining the quantized codebook data corresponding to a smallest relative reconstruction loss (RRL) by iterating over the second intermediate data.

14. The electronic device according to claim 13 , wherein the one or more processors, when obtaining the quantized codebook data corresponding to the smallest RRL by iterating over the second intermediate data, are further configured to implement an i-th iteration process, wherein i is an integer from 1 to K, K indicates a total iteration number, and 2 K is equal to N;

wherein in response to determining that i is equal to 1, a first iteration process comprises:

combining the N codebooks of the first intermediate data in pairs to obtain N/2 1 Level-1 codebook combinations, wherein each Level-1 codebook combination comprises C*C Level-1 element combinations, and each Level-1 element combination comprises 2 1 members;

for each Level-1 codebook combination, obtaining an RRL of each Level-1 element combination and the initial codebook indexes of remaining codebooks in the second intermediate data; and

for each Level-1 codebook combination, taking L Level-1 element combinations corresponding to smallest L RRLs as elements of the Level-1 codebook combination;

wherein in response to determining that i is an integer between 1 and K, the i-th iteration process comprises:

combining Level-(i−1) codebook combinations in pairs to obtain N/2 i Level-i codebook combinations, wherein each Level-i codebook combination comprises L*L Level-i element combinations, and each Level-i element combination comprises 2 i members;

for each Level-i codebook combination, obtaining an RRL of each Level-i element combination and the initial codebook indexes of remaining codebooks in the second intermediate data; and

for each Level-i codebook combination, taking L Level-i element combinations corresponding to smallest L RRLs as elements of the Level-i codebook combination; and

wherein in response to determining that i is equal to K, the K-th iteration process comprises:

combining Level-(K−1) codebook combinations in pairs to obtain a Level-K codebook combination, wherein the Level-K codebook combination comprises L*L Level-K element combinations, and each Level-K element combination comprises 2 K members;

obtaining an RRL of each Level-K element combination in the Level-K codebook combination; and

taking the Level-K element combination with a smallest RRL as the quantized codebook data.

15. The electronic device according to claim 12 , wherein the one or more processors are further configured to implement:

inputting the embedding into a codebook quantization model that is trained in advance, and

obtaining the quantized codebook data corresponding to a smallest relative reconstruction loss (RRL) outputted by the codebook quantization model after performing the multi-codebook quantization on the embedding.

16. The electronic device according to claim 15 , wherein the one or more processors are further configured to train the codebook quantization model, comprising:

obtaining the quantized codebook data and a first intermediate data outputted by the codebook quantization model by inputting an embedding sample into the codebook quantization model;

obtaining a reconstructed embedding outputted by a quantized-data reconstruction model by inputting the quantized codebook data into the quantized-data reconstruction model;

calculating an RRL based on the embedding sample and the reconstructed embedding, a uniform distribution loss based on the first intermediate data, and a consistency loss based on the first intermediate data and the quantized codebook data; and

completing training the codebook quantization model in response to determining at least one of followings: each of the RRL, the uniform distribution loss and the consistency loss meeting a requirement, or iteration times meeting a preset times threshold.

17. The electronic device according to claim 12 , wherein the loss comprises a categorical cross entropy, and the one or more processors, when calculating the loss based on the encoded data, the quantized codebook data and the text data in the training sample, are further configured to implement acts comprising:

obtaining a probability of each category in each codebook of the encoded data after a conversion;

obtaining a logarithm of each probability in each codebook;

calculating a sum of logarithms of all codebooks;

calculating a mean value of the sum; and

taking an inverse of the mean value as the categorical cross entropy.

18. The electronic device according to claim 12 , wherein the loss comprises a Recurrent Neural Network Transducer (RNN-T) loss function value, and the one or more processors, when calculating the loss based on the encoded data, the quantized codebook data and the text data in the training sample, are further configured to implement acts comprising:

obtaining a probability of each alignment scheme as enumerating alignment schemes between the speech data and the text data; and

obtaining a sum of probabilities corresponding to all the enumerating alignment schemes as the RNN-T loss function value.

19. A speech recognition device, comprising:

a memory and one or more processors;

wherein the memory is configured to store a computer program executable by the one or more processors; and

wherein the one or more processors are configured to execute the computer program in the memory to implement acts comprising:

acquiring speech data;

inputting the speech data into a preset speech recognition model; and

obtaining text data outputted by the speech recognition model;

wherein the speech recognition model is trained and implemented by adopting the training method according to claim 1 .

20. A non-transitory computer-readable storage medium, wherein an executable computer program in the storage medium, when being executed by one or more processors, implements the method according to claim 1 .

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (23)