Automatic speech recognition with multilingual scalability and low-resource adaptation
Systems and techniques that facilitate multilingual ASR machine learning models are provided. For example, one or more embodiments described herein can comprise a system, which can comprise a memory that can store computer executable components. The system can also comprise a processor, operably coupled to the memory that can execute the computer executable components stored in memory. The computer executable components can comprise an automatic speech recognition machine learning model that learns shared weights across one or more languages in a set of target languages; and a training component that trains the automatic speech recognition model.
1 . A system comprising:
a memory that stores computer executable components;
a processor that executes the computer executable components stored in the memory,
wherein the computer executable components comprise:
an automatic speech recognition machine learning model that learns shared weights across one or more languages in a set of target languages, the automatic speech recognition machine learning model comprising a projection layer having pretrained weights inherited from a previously trained model; and
a training component that trains the automatic speech recognition machine learning model, wherein the training comprises:
learning, using the projection layer, a mapping matrix having a plurality of rows corresponding respectively to languages of the set of target languages and columns corresponding to individual pretrained weights of the projection layer, the mapping matrix comprising rows of specialist scores for the one or more languages in the set of target languages;
generating a set of binary masks for the one or more languages in the set of target languages by thresholding the specialist scores of the mapping matrix to identify a subset of the pretrained weights associated with each language; and
adapting the automatic speech recognition machine learning model to the one or more languages of the set of target languages by selectively activating the subset of pretrained weights of the projection layer according to the generated binary masks while retaining remaining pretrained weights as shared weights across the set of target languages.
2 . The system of claim 1 , wherein the projection layer inherits the one or more pretrained weights from at least one of a QKV (query key value layer) layer or one or more previous projection layers of the automatic speech recognition machine learning model.
3 . The system of claim 1 , wherein the computer executable components further comprise:
a tuning component that tunes the automatic speech recognition machine learning model with an additional language added to the set of target languages.
4 . The system of claim 3 , wherein the tuning comprises:
locking parameters of the projection layer;
expanding the mapping matrix with an additional row for the additional language;
leaning specialist scores for the additional row of the mapping matrix;
generating a binary mask for the additional language; and
adapting the automatic speech recognition machine learning model to the additional language by applying the binary mask to the one or more pretrained weights of the projection layer.
5 . The system of claim 1 , wherein the specialist scores comprise a shared shape with the one or more pretrained weights.
6 . The system of claim 1 , wherein the automatic speech recognition machine learning model comprises a transformer model.
7 . The system of claim 1 , wherein the shared weights represent phonetic and syntactic similarities between languages of the set of target languages.
8 . A computer-implemented method comprising:
training, by a system operatively coupled to a processor, an automatic speech recognition machine learning model to learn shared weights across one or more languages in a set of target languages, the automatic speech recognition machine learning model comprising a projection layer having pretrained weights inherited from a previously trained speech recognition model, wherein the training comprises:
learning, by the system, using the projection layer of the automatic speech recognition machine learning model, a mapping matrix having a plurality of rows corresponding respectively to languages of the set of target languages and columns corresponding to projection weights of the projection layer, the mapping matrix comprising rows of specialist scores for the one or more languages in the set of target languages;
generating, by the system, a set of binary masks for the one or more languages in the set of target languages based on the specialist scores of the mapping matrix, each binary mask identifying a subset of projection weights associated with a respective language; and
adapting, by the system, the automatic speech recognition machine learning model to the one or more languages of the set of target languages by applying the binary mask corresponding to a selected language to modulate outputs of the projection layer using the subset of projection weights identified by the binary mask while maintaining remaining projection weights as shared weights across the set of target languages.
9 . The computer-implemented method of claim 8 , wherein the projection layer inherits the one or more pretrained weights from at least one of a QKV (query key value layer) layer or one or more previous projection layers of the automatic speech recognition machine learning model.
10 . The computer-implemented method of claim 8 , further comprising tuning, by the system, the automatic speech recognition machine learning model with an additional language added to the set of target languages.
11 . The computer-implemented method of claim 10 , wherein the tuning comprises:
locking, by the system, parameters of the projection layer;
expanding, by the system, the mapping matrix with an additional row for the additional language;
leaning, by the system, specialist scores for the additional row of the mapping matrix;
generating, by the system, a binary mask for the additional language; and
adapting, by the system, the automatic speech recognition machine learning model to the additional language by applying the binary mask to the one or more pretrained weights of the projection layer.
12 . The computer-implemented method of claim 8 , wherein the specialist scores comprise a shared shape with the one or more pretrained weights.
13 . The computer-implemented method of claim 8 , wherein the automatic speech recognition machine learning model comprises a transformer model.
14 . The computer-implemented method of claim 8 , wherein the shared weights represent phonetic and syntactic similarities between languages of the set of target languages.
15 . A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:
train, by the processor, an automatic speech recognition machine learning model to learn shared weights across one or more languages in a set of target languages, the automatic speech recognition machine learning model comprising a projection layer having pretrained weights inherited from a previously trained speech recognition model, wherein the training comprises:
learning, by the processor, using the projection layer of the automatic speech recognition machine learning model, a mapping matrix having a plurality of rows corresponding respectively to languages of the set of target languages and columns corresponding to projection weights of the projection layer, the mapping matrix comprising rows of specialist scores for the one or more languages in the set of target languages;
generating, by the processor, a set of binary masks for the one or more languages in the set of target languages based on the specialist scores of the mapping matrix, each binary mask representing a language-specific selection vector corresponding to the projection weights of the projection layer; and
adapting, by the processor, the automatic speech recognition machine learning model to the one or more languages of the set of target languages by selectively applying, by the program instructions, the binary masks to the pretrained weights of the projection layer to enable language-specific specialization of the projection layer while retaining remaining pretrained weights as shared parameters across the set of target languages.
16 . The computer program product of claim 15 , wherein the projection layer inherits the one or more pretrained weights from at least one of a QKV (query key value layer) layer or one or more previous projection layers of the automatic speech recognition machine learning model.
17 . The computer program product of claim 15 , wherein the program instructions are further executable to cause the processor to:
tune, by the processor, the automatic speech recognition machine learning model with an additional language added to the set of target languages.
18 . The computer program product of claim 17 , wherein the tuning comprises:
locking, by the processor, parameters of the projection layer;
expanding, by the processor, the mapping matrix with an additional row for the additional language;
leaning, by the processor, specialist scores for the additional row of the mapping matrix;
generating, by the processor, a binary mask for the additional language; and
adapting, by the processor, the automatic speech recognition machine learning model to the additional language by applying the binary mask to the one or more pretrained weights of the projection layer.
19 . The computer program product of claim 15 , wherein the specialist scores comprise a shared shape with the one or more pretrained weights.
20 . The computer program product of claim 15 , wherein the shared weights represent phonetic and syntactic similarities between languages of the set of target languages.