Systems and methods for translation evaluation
Systems and methods for translation evaluation are provided. One or more aspects of the systems and methods includes receiving a source text, a context identifier for the source text, and a translation text, wherein the source text comprises text from a software application and the context identifier specifies a context of the source text within the software application; generating a source text representation and a translation text representation based on the source text, the context identifier, and the translation text using an encoder of a machine learning model; and generating translation quality information based on the source text representation and the translation text representation using a decoder of the machine learning model.
1 . A method comprising:
receiving a source text, a context identifier for the source text, and a translation text, wherein the source text comprises text displayed in a user interface of a software application and the context identifier comprises text that describes the user interface;
generating a sequence of contextual embeddings by tokenizing the context identifier to obtain a token sequence and performing contextual embedding on the token sequence to obtain the sequence of contextual embeddings;
generating a context vector based on the sequence of contextual embeddings, wherein the context vector represents the context identifier in a vector space;
generating a source text representation based on the source text and the context vector using an encoder of a machine learning model, wherein the source text representation comprises a representation of the source text and a visual context of the source text based on the context identifier;
generating a translation text representation based on the translation text using the encoder; and
generating a translation quality prediction score for the translation text based on the source text representation and the translation text representation using a decoder of the machine learning model, wherein the translation quality prediction score accounts for the visual context of the source text within the user interface.
2 . The method of claim 1 , further comprising:
generating a source text embedding and a translation text embedding based on the source text and the translation text, respectively, wherein the source text representation and the translation text representation are based on the source text embedding and the translation text embedding, respectively.
3 . The method of claim 2 , further comprising:
identifying a source language of the source text and a translation language of the translation text; and
selecting a source language model based on the source language and a translation language model based on the translation language, wherein the source text embedding and the translation text embedding are based on the source language model and the translation language model, respectively.
4 . The method of claim 2 , further comprising:
removing markup tags from the source text and the translation text to obtain a pre-processed source text and a pre-processed translation text, respectively, wherein the source text embedding is based on the pre-processed source text and the translation text embedding is based on the pre-processed translation text.
5 . The method of claim 2 , further comprising:
replacing localization terms in the source text and the translation text with localization placeholders to obtain a pre-processed source text and a pre-processed translation text, respectively, wherein the source text embedding is based on the pre-processed source text and the translation text embedding is based on the pre-processed translation text.
6 . The method of claim 2 , further comprising:
generating a source text attention mask and a translation text attention mask based on the source text and the translation text, respectively, wherein the source text representation and the translation text representation are based on the source text attention mask and the translation text attention mask, respectively.
7 . The method of claim 2 , further comprising:
generating a hidden source text representation based on the source text embedding and the context vector using a recurrent neural network of the encoder; and
generating a hidden translation text representation based on the translation text embedding using the recurrent neural network.
8 . The method of claim 7 , further comprising:
generating the source text representation and the translation text representation based on the hidden source text representation and the hidden translation text representation, respectively, using a cross-attention component of the encoder.
9 . The method of claim 1 , further comprising:
combining the sequence of contextual embeddings to obtain the context vector.
10 . The method of claim 1 , further comprising:
generating a cross-lingual textual entailment (CLTE) feature vector representing a plurality of structural features of the source text and the translation text, wherein the translation quality prediction information is based on the CLTE feature vector.
11 . The method of claim 10 , further comprising:
identifying a phrase dictionary between a source language of the source text and a target language of the translation text; and
computing dictionary features based on the phrase dictionary, wherein the CLTE feature vector includes the dictionary features.
12 . The method of claim 1 , further comprising:
generating a binary class label for the translation text based on the source text representation and the translation text representation, wherein the translation quality prediction score includes the binary class label.
13 . The method of claim 1 , further comprising:
generating a real-valued score for the translation text based on the source text representation and the translation text representation, wherein the translation quality prediction score includes the real-valued score.
14 . The method of claim 1 , further comprising:
generating a revised translation text for the source text based on the translation quality prediction score.
15 . A method comprising:
receiving training data including a source text, a context identifier for the source text, a translation text, and ground-truth translation quality information, wherein the source text comprises text displayed in a user interface of a software application and the context identifier comprises text that describes the user interface;
generating a sequence of contextual embeddings by tokenizing the context identifier to obtain a token sequence and performing contextual embedding on the token sequence to obtain the sequence of contextual embeddings;
generating a context vector based on the sequence of contextual embeddings, wherein the context vector represents the context identifier in a vector space;
generating a source text representation based on the source text and the context vector using an encoder of a machine learning model, wherein the source text representation comprises a representation of the source text and a visual context of the source text based on the context identifier;
generating a translation text representation based on the translation text using the encoder;
generating a translation quality prediction score for the translation text based on the source text representation and the translation text representation using a decoder of the machine learning model, wherein the translation quality prediction score accounts for the visual context of the source text within the user interface;
computing a loss function based on the predicted translation quality information and the ground-truth translation quality information; and
updating parameters of the machine learning model based on the loss function.
16 . A system comprising:
a memory component; and
a processing device coupled to the memory component, the processing device configured to perform operations comprising:
receiving a source text, a context identifier for the source text, and a translation text, wherein the source text comprises text displayed in a user interface of a software application and the context identifier comprises text that describes the user interface;
generating a sequence of contextual embeddings by tokenizing the context identifier to obtain a token sequence and performing contextual embedding on the token sequence to obtain the sequence of contextual embeddings;
generating a context vector based on the sequence of contextual embeddings, wherein the context vector represents the context identifier in a vector space;
generating a source text representation based on the source text and the context vector using an encoder of a machine learning model, wherein the source text representation comprises a representation of the source text and a visual context of the source text based on the context identifier;
generating a translation text representation based on the translation text using the encoder; and
generating a translation quality prediction score for the translation text based on the source text representation and the translation text representation using a decoder of the machine learning model, wherein the translation quality prediction score accounts for the visual context of the source text within the user interface.
17 . The system of claim 16 , further comprising:
a cross-lingual textual entailment (CLTE) component configured to generate a CLTE feature vector representing a plurality of structural features of the source text and the translation text, wherein the translation quality prediction score is based on the CLTE feature vector.
18 . The system of claim 16 , wherein:
the encoder includes a recurrent neural network and a cross-attention module.
19 . The system of claim 16 , wherein:
the decoder includes a regressor network and a classifier network.