IP Library › Granted Patent US 12664978
Granted Patent B2
US 12664978 · App. 18/078,782 · Granted Jun 23, 2026

Federated knowledge distillation on an encoder of a global ASR model and/or an encoder of a client ASR model

Inventors: Ehsan Amid (Mountain View, CA); Rajiv Mathews (Sunnyvale, CA); Shankar Kumar (New York, NY); Jared Lichtarge (Brooklyn, NY); Mingqing Chen (Saratoga, CA); Tien-Ju Yang (Mountain View, CA); Yuxin Ding (San Francisco, CA)
Assignee: GOOGLE LLC
G10L15/16G10L15/063
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12664978
App. No.
18/078,782
Granted
Jun 23, 2026
Kind
B2
Abstract

Information can be distilled from a global automatic speech recognition (ASR) model to a client ASR model. Many implementations include using an RNN-T model as the ASR model, where the global ASR model includes a global encoder, a joint network, a prediction network, and where the client ASR model includes a client encoder, the joint network, and the prediction network. Various implementations include using principal component analysis (PCA) while training the global ASR model to learn a mean vector and a set of principal components corresponding to the global ASR model. Additional or alternative implementations include training the client ASR model to generate one or more predicted coefficients of the global ASR model.

Claims (64)

1 . A method implemented by one or more processors, the method comprising:

distilling information from a global automatic speech recognition (“ASR”) model to generate a client ASR model, wherein the global ASR model includes a global encoder, a prediction model, and a joint network, wherein the client ASR model includes a client encoder that is smaller than the global encoder of the global ASR model, the prediction model of the global ASR model, and the joint network of the global ASR model, and wherein distilling the global ASR model to generate the client ASR model comprises:

processing a set of training instances using principal component analysis (“PCA”) to generate (a) a mean vector for the set of training instances and (b) a set of principal directions for the set of training instances;

for each of a plurality of training instances in the set of training instances and until one or more conditions are satisfied:

selecting a given training instance, wherein the given training instance includes an instance of audio data capturing a spoken utterance;

processing the instance of audio data capturing the spoken utterance using the client ASR model to generate one or more predicted coefficients corresponding to the given training instance;

processing the instance of audio data capturing the spoken utterance using the global ASR model to generate global output;

generating a loss based on the global output generated based on processing the instance of audio data using the global ASR model, the one or more predicted coefficients generated based on processing the instance of audio data using the client ASR model, the mean vector for the set of training instances generated based on processing the set of training instances using PCA, and the set of principal directions for the set of training instances generated based on processing the set of training instances using PCA; and

updating one or more portions of the client encoder based on comparing the loss and the one or more predicted coefficients.

2 . The method of claim 1 , further comprising:

processing a further instance of audio data capturing a further spoken utterance using the global ASR model to generate a global text representation of the spoken utterance;

processing the further instance of audio data using the client ASR model to generate a client text representation of the spoken utterance; and

updating one or more portions of the client encoder based on comparing the global text representation of the spoken utterance and the client text representation of the spoken utterance.

3 . The method of claim 2 , wherein updating the one or more portions of the client encoder based on comparing the global text representation of the spoken utterance and the client text representation of the spoken utterance comprises:

freezing the joint network of the client ASR model and freezing the prediction network of the client ASR model.

4 . The method of claim 1 , wherein the global ASR model is a recurrent neural network transformer (“RNN-T”) and wherein the client ASR model is an additional RNN-T.

5 . The method of claim 1 , wherein processing the set of training instances using PCA to generate (a) the mean vector for the set of training instances and (b) the set of principal directions for the set of training instances comprises:

processing the set of training instances using Bregman PCA to generate (a) the mean vector for the set of training instances and (b) the set of principal directions for the set of training instances.

6 . The method of claim 1 , wherein the client ASR model is stored locally at a client device and wherein the global ASR model is stored at a server remote from the client device.

7 . The method of claim 6 , wherein storage of the global encoder takes a first value of memory, wherein storage of the client encoder takes a second value of memory, and wherein the first value of memory is greater than the second value of memory.

8 . The method of claim 1 , wherein the global ASR model is initially trained using a set of non-private training data.

9 . The method of claim 1 , further comprising:

distilling information from the client ASR model to the global ASR model, wherein distilling information from the client ASR model to the global ASR model comprises:

processing an additional instance of audio data capturing an additional spoken utterance using the client ASR model to generate one or more additional predicted coefficients;

generating additional global output based on processing the one or more additional predicted coefficients, the mean vector, and the set of principal directions using the global ASR model;

generating an additional loss based on the one or more additional predicted coefficients, the additional global output, the mean vector, and the set of principal directions; and

updating one or more portions of the global encoder based on comparing the additional loss and the one or more additional predicted coefficients.

10 . A method implemented by one or more processors, the method comprising:

distilling information from a client automatic speech recognition (“ASR”) model to a global ASR model, wherein the client ASR model includes a client encoder, a prediction model, and a joint network, wherein the global ASR model includes a global encoder that is larger than the client encoder of the client ASR model, the prediction model, and the joint network, and wherein distilling information from the client ASR model to the global ASR model comprises:

processing an instance of audio data capturing a spoken utterance using the client ASR model to generate one or more predicted coefficients corresponding to the spoken utterance;

generating global output based on processing the one or more predicted coefficients, a mean vector of the global ASR model, a set of principal directions of the global ASR model using the global ASR model, where the mean vector of the global ASR model and the set of principal directions for the global ASR model are generated using principal component analysis (“PCA”);

generating a loss based on the one or more predicted coefficients generated based on processing the instance of audio data using the client ASR model, the global output, the mean vector of the global ASR model generated using PCA, and the set of principal directions of the global ASR model generated using PCA; and

updating one or more portions of the global encoder based on comparing the loss and the one or more predicted coefficients.

11 . The method of claim 10 , further comprising:

distilling information from an additional client ASR model to the global ASR model, wherein distilling information from the additional client ASR model to the global ASR model comprises:

processing an additional instance of audio data capturing an additional spoken utterance using the additional client ASR model to generate one or more additional predicted coefficients corresponding to the additional spoken utterance;

generating additional global output based on processing the one or more additional predicted coefficients, the mean vector of the global ASR model, and the set of principal directions of the global ASR model using the global ASR model;

generating an additional loss based on the one or more additional predicted coefficients, the additional global output, the mean vector of the global ASR model, and the set of principal directions of the global ASR model; and

updating one or more portions of the global encoder based on comparing the additional loss and the one or more additional predicted coefficients.

12 . The method of claim 10 , wherein updating the one or more portions of the global encoder based on comparing the loss and the one or more predicted coefficients comprises:

freezing the joint network of the global ASR model and freezing the prediction network of the global ASR model.

13 . The method of claim 10 , wherein the global ASR model is a recurrent neural network transformer (“RNN-T”) and wherein the client ASR model is an additional RNN-T.

14 . The method of claim 10 , wherein the mean vector of the global ASR model and the set of principal directions of the global ASR model are generated using Bregman PCA.

15 . The method of claim 10 , wherein the client ASR model is stored locally at a client device and wherein the global ASR model is stored at a server remote from the client device.

16 . A system comprising:

memory storing instructions;

one or more processors operable to execute the instructions to:

distilling information from a global automatic speech recognition (“ASR”) model to generate a client ASR model, wherein the global ASR model includes a global encoder, a prediction model, and a joint network, wherein the client ASR model includes a client encoder that is smaller than the global encoder of the global ASR model, the prediction model, and the joint network, and wherein distilling the global ASR model to generate the client ASR model comprises:

processing a set of training instances using principal component analysis (“PCA”) to generate (a) a mean vector for the set of training instances and (b) a set of principal directions for the set of training instances;

for each of a plurality of training instances in the set of training instances and until one or more conditions are satisfied:

selecting a given training instance, wherein the given training instance includes an instance of audio data capturing a spoken utterance;

processing the instance of audio data capturing the spoken utterance using the client ASR model to generate one or more predicted coefficients corresponding to the given training instance;

processing the instance of audio data capturing the spoken utterance using the global ASR model to generate global output;

generating a loss based on the global output generated based on processing the instance of audio data using the global ASR model, the one or more predicted coefficients generated based on processing the instance of audio data using the client ASR model, the mean vector for the set of training instances generated based on processing the set of training instances using PCA, and the set of principal directions for the set of training instances generated based on processing the set of training instances using PCA; and

updating one or more portions of the client encoder based on comparing the loss and the one or more predicted coefficients.

17 . The system of claim 16 , wherein one or more of the processors are further operable to execute the instructions to:

processing a further instance of audio data capturing a further spoken utterance using the global ASR model to generate a global text representation of the spoken utterance;

processing the further instance of audio data using the client ASR model to generate a client text representation of the spoken utterance; and

updating one or more portions of the client encoder based on comparing the global text representation of the spoken utterance and the client text representation of the spoken utterance.

18 . The system of claim 17 , wherein updating the one or more portions of the client encoder based on comparing the global text representation of the spoken utterance and the client text representation of the spoken utterance comprises:

freezing the joint network of the client ASR model and freezing the prediction network of the client ASR model.

19 . The system of claim 16 , wherein the global ASR model is a recurrent neural network transformer (“RNN-T”) and wherein the client ASR model is an additional RNN-T.

20 . The system of claim 16 , wherein processing the set of training instances using PCA to generate (a) the mean vector for the set of training instances and (b) the set of principal directions for the set of training instances comprises:

processing the set of training instances using Bregman PCA to generate (a) the mean vector for the set of training instances and (b) the set of principal directions for the set of training instances.