Private language model adaptation for speech recognition

1. A method comprising, by a first computing device:

accessing a decoded hypothesis corresponding to an utterance;

computing a predicted probability of observing tokens in the decoded hypothesis by having a local first machine-learning model process the decoded hypothesis;

computing, for the tokens in the decoded hypothesis, confidence scores by having a second machine-learning model process the decoded hypothesis, wherein the confidence scores indicate a degree of confidence for the tokens to be observed at their position;

calculating a loss for the computed predicted probabilities of observing tokens in the decoded hypothesis based on the computed confidence scores; and

updating parameters of the local first machine-learning model based on the calculated loss.

2. The method of claim 1 , wherein the local first machine-learning model is a neural network language model (NNLM) used for an automatic speech recognition (ASR) system.

3. The method of claim 1 , wherein the local first machine-learning model is a copy of a global first machine-learning model that is pre-trained on a server using a corpus of training data.

4. The method of claim 3 , further comprising:

sending the updated parameters of the local first machine-learning model to the server, wherein the server updates other parameters of the global first machine-learning model based on the updated parameters.

5. The method of claim 4 , wherein the server updates the other parameters of the global first machine-learning model based on one or more updated parameters of local first machine-learning models received from second computing devices.

6. The method of claim 5 , wherein a FedAdam optimizer is used for updating the parameters of the global first machine-learning model.

7. The method of claim 5 , wherein the server applies a weight assigned to a computing device when updating the parameters of the global first machine-learning model based on updated parameters received from the computing device.

8. The method of claim 5 , further comprising:

receiving, from the server, the updated parameters of the global first machine-learning model; and

updating the parameters of the local first machine-learning model using the received updated parameters of the global first machine-learning model.

9. The method of claim 1 , wherein the decoded hypothesis corresponding to the utterance is generated by an automatic speech recognition (ASR) based on acoustic feature vectors extracted from speech of a user associated with the first computing device.

10. The method of claim 1 , wherein the loss is a cross entropy loss.

11. The method of claim 10 , wherein token-level weights determined based on the computed confidence scores are applied when the loss is calculated.

12. The method of claim 10 , wherein an utterance-level weight determined by averaging confidence scores for transcribed tokens corresponding to the utterance is applied when the loss is calculated.

13. The method of claim 1 , wherein a mini-batch stochastic gradient descent (SGD) is used as a local optimizer when updating the parameters of the local first machine-learning model.

14. The method of claim 1 , wherein the second machine-learning model is a pre-trained confidence classifier.

15. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

access a decoded hypothesis corresponding to an utterance;

compute a predicted probability of observing tokens in the decoded hypothesis by having a local first machine-learning model process the decoded hypothesis;

compute, for the tokens in the decoded hypothesis, confidence scores by having a second machine-learning model process a sequence of transcribed tokens, wherein the confidence scores indicate a degree of confidence for the tokens to be observed at their position;

calculate a loss for the computed predicted probabilities of observing the tokens in the decoded hypothesis based on the computed confidence scores; and

update parameters of the local first machine-learning model based on the calculated loss.

16. The media of claim 15 , wherein the local first machine-learning model is a neural network language model (NNLM) used for an automatic speech recognition (ASR) system.

17. The media of claim 15 , wherein the local first machine-learning model is a copy of a global first machine-learning model that is pre-trained on a server using a corpus of training data.

18. The media of claim 17 , wherein the software is further operable when executed to:

send the updated parameters of the local first machine-learning model to the server, wherein the server updates parameters of the global first machine-learning model based on the updated parameters of the local first machine-learning model.

19. The media of claim 18 , wherein the server updates the parameters of the global first machine-learning model based on other updated parameters of local first machine-learning models received from second computing devices.

20. A system comprising: one or more processors; and a non-transitory memory coupled to the one or more processors comprising instructions executable by the one or more processors, the one or more processors operable when executing the instructions to:

access a decoded hypothesis corresponding to an utterance;

compute a predicted probability of observing the tokens in the decoded hypothesis by having a local first machine-learning model process the decoded hypothesis;

compute, for the tokens in the decoded hypothesis, confidence scores by having a second machine-learning model process the sequence of transcribed tokens, wherein the confidence scores indicate a degree of confidence for the tokens to be observed at their position;

calculate a loss for the computed predicted probabilities of observing the tokens in the decoded hypothesis based on the computed confidence scores; and

update parameters of the local first machine-learning model based on the calculated loss.