Fine-tuning language models using target language vocabulary and parallel data for machine translation
Techniques for fine-tuning language models using target language vocabulary and parallel data for machine translation are disclosed. In an example method, a computing device receives one or more words in a source language. The computing device translates the one or more words to a target language using a fine-tuned language model. The language model is fine-tuned by accessing a foundation language model pre-trained using source language training data and having a source language vocabulary. A set of tokens is added to the model representing a target language vocabulary. The model is further trained using target language training data. The model is further trained using parallel training data including a number of parallel elements each parallel element including words in the source language and the corresponding words in the target language. The computing device outputs the translated one or more words in the target language.
1 . A method, comprising:
receiving one or more words in a source language;
translating the one or more words to a target language using a fine-tuned language model, wherein the language model is fine-tuned by:
accessing a foundation language model, the foundation language model being pre-trained using a first quantity of source language training data and having a source language vocabulary comprising a first plurality of tokens based on the source language;
adding a second plurality of tokens to the foundation language model, the second plurality of tokens representing a target language vocabulary according to the target language;
training the foundation language model using a second quantity of target language training data; and
training the foundation language model using a third quantity of parallel training data, wherein the third quantity of parallel training data comprises a plurality of parallel elements, each parallel element comprising one or more words in the source language and a corresponding one or more words in the target language; and
outputting the one or more words in the target language.
2 . The method of claim 1 , wherein adding the second plurality of tokens to the foundation language model comprises:
generating the second plurality of tokens based on a set of words in the target language;
adding the second plurality of tokens to the source language vocabulary; and
resizing an embedding matrix associated with the language model using a sum of a first count of the first plurality of tokens and a second count of the second plurality of tokens.
3 . The method of claim 1 , wherein the foundation language model is a large language model based on a transformer architecture.
4 . The method of claim 1 , wherein the plurality of parallel elements includes at least 1 million elements.
5 . The method of claim 1 , wherein the plurality of parallel elements includes at least 10 million elements.
6 . The method of claim 1 , wherein:
the third quantity of parallel training data is generated using a manual translation process from the source language to the target language; and
the third quantity of parallel training data is filtered using a neural network trained to classify an accuracy of, for each parallel element, the one or more words in the target language.
7 . The method of claim 1 , wherein the language model is further fine-tuned by:
adding a third plurality of tokens to the foundation language model, the third plurality of tokens representing a second target language vocabulary according to a second target language;
training the foundation language model using a fourth quantity of second target language training data; and
training the foundation language model using a fifth quantity of parallel training data, wherein the fifth quantity of parallel training data comprises a second plurality of parallel elements, each parallel element comprising one or more words in the source language and a corresponding one or more words in the second target language.
8 . The method of claim 7 , further comprising outputting the one or more words in the second target language.
9 . A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:
receiving one or more words in a source language;
translating the one or more words to a target language using a fine-tuned language model, wherein the language model is fine-tuned by:
accessing a foundation language model, the foundation language model being pre-trained using a first quantity of source language training data and having a source language vocabulary comprising a first plurality of tokens based on the source language;
adding a second plurality of tokens to the foundation language model, the second plurality of tokens representing a target language vocabulary according to the target language;
training the foundation language model using a second quantity of target language training data; and
training the foundation language model using a third quantity of parallel training data, wherein the third quantity of parallel training data comprises a plurality of parallel elements, each parallel element comprising one or more words in the source language and a corresponding one or more words in the target language; and
outputting the one or more words in the target language.
10 . The non-transitory computer-readable medium of claim 9 , wherein adding the second plurality of tokens to the foundation language model comprises:
generating the second plurality of tokens, the second plurality of tokens comprising a set including words, portions of words, or characters based on a set of words in the target language vocabulary;
adding the second plurality of tokens to the source language vocabulary; and
resizing an embedding matrix associated with the language model using a sum of a first count of the first plurality of tokens and a second count of the second plurality of tokens.
11 . The non-transitory computer-readable medium of claim 9 , wherein the foundation language model is a large language model based on a transformer architecture.
12 . The non-transitory computer-readable medium of claim 9 , wherein the plurality of parallel elements includes at least 20 million elements.
13 . The non-transitory computer-readable medium of claim 9 , wherein the second plurality of tokens includes at least 10 thousand tokens.
14 . The non-transitory computer-readable medium of claim 9 , wherein the parallel elements each include a sentence in the source language and the corresponding sentence translated into the target language.
15 . The non-transitory computer-readable medium of claim 9 , wherein the parallel elements each include a sentence in the source language and the sentence in the target language.
16 . A system comprising:
one or more processors; and
one or more computer-readable storage media storing instructions which, when executed by the one or more processors, cause the one or more processors to perform operations including:
receiving one or more words in a source language;
translating the one or more words to a target language using a fine-tuned language model, wherein the language model is fine-tuned by:
accessing a foundation language model, the foundation language model being pre-trained using a first quantity of source language training data and having a source language vocabulary comprising a first plurality of tokens based on the source language;
adding a second plurality of tokens to the foundation language model, the second plurality of tokens representing a target language vocabulary according to the target language;
training the foundation language model using a second quantity of target language training data; and
training the foundation language model using a third quantity of parallel training data, wherein the third quantity of parallel training data comprises a plurality of parallel elements, each parallel element comprising one or more words in the source language and a corresponding one or more words in the target language; and
outputting the one or more words in the target language.
17 . The system of claim 16 , wherein:
generating the second plurality of tokens, the second plurality of tokens comprising a set including words, portions of words, or characters based on the set of words in the target language;
adding the second plurality of tokens to the source language vocabulary; and
resizing an embedding matrix associated with the language model using a sum of a first count of the first plurality of tokens and a second count of the second plurality of tokens.
18 . The system of claim 16 , wherein the foundation language model is a large language model based on a transformer architecture.
19 . The system of claim 16 , wherein the plurality of parallel elements includes at least 100 million elements.
20 . The system of claim 16 , wherein the language model is further fine-tuned by:
adding a third plurality of tokens to the foundation language model, the third plurality of tokens representing a second target language vocabulary according to a second target language;
training the foundation language model using a fourth quantity of second target language training data; and
training the foundation language model using a fifth quantity of parallel training data, wherein the fifth quantity of parallel training data comprises a second plurality of parallel elements, each parallel element comprising one or more words in the source language and a corresponding one or more words in the second target language,
and further comprising the operation of outputting the one or more words in the second target language.