Systems and methods for continual learning for end to-end automatic speech recognition
In some aspects, the techniques described herein relate to a method including: providing, to a parallel model training platform, a plurality of domain datasets; training, by the parallel model training platform, a plurality of generalist models in parallel, wherein each generalist model of the plurality of generalist models is trained in parallel using a corresponding one of the plurality of domain datasets, and wherein training the plurality of generalist models in parallel generates a corresponding expert model for each generalist model in the plurality of generalist models; executing, by the parallel model training platform, a model parameter averaging process, wherein the model parameter averaging process take each corresponding expert model as input; and generating, by the parallel model training platform and as output of the model parameter averaging process, an average-of-domain-experts (AoDE) model.
1 . A method comprising:
providing, to a parallel model training platform, a plurality of domain datasets;
training, by the parallel model training platform, a plurality of generalist models in parallel, wherein each generalist model of the plurality of generalist models is trained in parallel using a corresponding one of the plurality of domain datasets, and wherein training the plurality of generalist models in parallel generates a corresponding expert model for each generalist model in the plurality of generalist models, wherein each of the plurality of generalist models is a copy of a single generalist model;
executing, by the parallel model training platform, a model parameter averaging process, wherein the model parameter averaging process take each corresponding expert model as input and includes an equal-weight linear interpolation of model parameters of each corresponding expert model; and
generating, by the parallel model training platform and as output of the model parameter averaging process, an average-of-domain-experts (AoDE) model.
2 . The method of claim 1 , comprising:
executing, by the model parameter averaging process, a non-linear interpolation of model parameters of each corresponding expert model.
3 . The method of claim 1 , wherein one of the plurality of domain datasets is a random sampling of a larger domain dataset.
4 . The method of claim 1 , wherein one of the plurality of domain datasets is divided into time segments that do not exceed a threshold value.
5 . A system comprising at least one computer including a processor and a memory, wherein the at least one computer is configured to:
provide, to a parallel model training platform, a plurality of domain datasets;
train, by the parallel model training platform, a plurality of generalist models in parallel, wherein each generalist model of the plurality of generalist models is trained in parallel using a corresponding one of the plurality of domain datasets, and wherein training the plurality of generalist models in parallel generates a corresponding expert model for each generalist model in the plurality of generalist models, wherein each of the plurality of generalist models is a copy of a single generalist model;
execute, by the parallel model training platform, a model parameter averaging process, wherein the model parameter averaging process take each corresponding expert model as input and includes an equal-weight linear interpolation of model parameters of each corresponding expert model; and
generate, by the parallel model training platform and as output of the model parameter averaging process, an average-of-domain-experts (AoDE) model.
6 . The system of claim 5 , wherein the at least one computer is configured to:
execute, by the model parameter averaging process, a non-linear interpolation of model parameters of each corresponding expert model.
7 . The system of claim 5 , wherein one of the plurality of domain datasets is a random sampling of a larger domain dataset.
8 . The system of claim 5 , wherein one of the plurality of domain datasets is divided into time segments that do not exceed a threshold value.
9 . A non-transitory computer readable storage medium, including instructions stored thereon, which instructions, when read and executed by one or more computer processors, cause the one or more computer processors to perform steps comprising:
providing, to a parallel model training platform, a plurality of domain datasets;
training, by the parallel model training platform, a plurality of generalist models in parallel, wherein each generalist model of the plurality of generalist models is trained in parallel using a corresponding one of the plurality of domain datasets, and wherein training the plurality of generalist models in parallel generates a corresponding expert model for each generalist model in the plurality of generalist models, wherein each of the plurality of generalist models is a copy of a single generalist model;
executing, by the parallel model training platform, a model parameter averaging process, wherein the model parameter averaging process takes each corresponding expert model as input and includes an equal-weight linear interpolation of model parameters of each corresponding expert model; and
generating, by the parallel model training platform and as output of the model parameter averaging process, an average-of-domain-experts (AoDE) model.
10 . The non-transitory computer readable storage medium of claim 9 , comprising:
executing, by the model parameter averaging process, a non-linear interpolation of model parameters of each corresponding expert model.
11 . The non-transitory computer readable storage medium of claim 9 , wherein one of the plurality of domain datasets is a random sampling of a larger domain dataset.