System and method for machine learning a confidence metric for machine translation
View Patent ↗A machine translation system is trained to generate confidence scores indicative of a quality of a translation result. A source string is translated with a machine translator to generate a target string. Features indicative of translation operations performed are extracted from the machine translator. A trusted entity-assigned translation score is obtained and is indicative of a trusted entity-assigned translation quality of the translated string. A relationship between a subset of the extracted features and the trusted entity-assigned translation score is identified.
1. A method of training a machine translation system to generate confidence scores indicative of a quality of a translation result, comprising:
translating a source string with a machine translator to generate a target string;
extracting features from the machine translator, indicative of performance of translation steps in the machine translator;
obtaining a trusted entity-assigned translation score indicative of a trusted entity-assigned translation quality of the target string;
identifying a relationship between a subset of the extracted features and the trusted entity-assigned translation score by:
selecting a subset of the features that has a desired influence on a correlation coefficient, the correlation coefficient being indicative of a correlation between the subset of extracted features and the trusted entity-assigned score; and
identifying a constant and coefficients for each feature in the subset that together produced the desired influence on the correlation coefficient.
2. The method of claim 1 wherein translating a source string comprises:
translating a plurality of source strings to generate a plurality of corresponding target strings.
3. The method of claim 2 wherein extracting features comprises:
extracting a plurality of features associated with each of the plurality of target strings.
4. The method of claim 1 wherein translating includes:
parsing the source string into a source intermediate linguistic structure indicative of a meaning of the source string and wherein extracting features comprises:
extracting one or more features indicative of a quality of parsing.
5. The method of claim 4 wherein extracting one or more features indicative of a quality of parsing comprises:
extracting a feature indicative of whether parsing generated a fitted parse as the intermediate structure.
6. The method of claim 1 wherein extracting features comprises:
extracting a feature indicative of a size of the source string.
7. The method of claim 4 wherein translating includes translating the source intermediate linguistic structure to a target intermediate linguistic structure and wherein extracting features comprises:
extracting one or more features indicative of a quality of translating the source intermediate linguistic structure into the target intermediate linguistic structure.
8. The method of claim 7 wherein translating the source intermediate linguistic structure comprises:
identifying mappings, in a mapping database, that map portions of the source intermediate linguistic structure to portions of the target intermediate linguistic structure.
9. The method of claim 8 wherein extracting one or more features indicative of a quality of translating the source intermediate linguistic structure comprises:
extracting a feature indicative of a number of identified mappings.
10. The method of claim 8 wherein extracting one or more features indicative of a quality of translating the source intermediate linguistic structure comprises:
extracting a feature indicative of a size of identified mappings.
11. The method of claim 8 wherein extracting one or more features indicative of a quality of translating the source intermediate linguistic structure comprises:
extracting a feature indicative of a frequency with which mappings where identified.
12. The method of claim 8 wherein each mapping has an associated confidence score and wherein extracting one or more features indicative of a quality of translating the source intermediate linguistic structure comprises:
extracting a feature indicative of the confidence scores associated with the identified mappings.
13. The method of claim 8 wherein translating the source intermediate linguistic structure comprises:
translating portions of the source intermediate linguistic structure using another translation device other than the mapping database.
14. The method of claim 1 wherein extracting features comprises:
extracting a feature indicative of the translation device used.
15. The method of claim 14 wherein extracting features comprises:
extracting a feature indicative of an amount of the source string translated with another translation device.
16. The method of claim 14 wherein extracting features comprises:
extracting a feature indicative of a type of words translated with another translation device.
17. The method of claim 13 wherein some words in the source intermediate linguistic structure remain untranslated and wherein extracting one or more features indicative of a quality of translating the source intermediate linguistic structure comprises:
extracting a feature indicative of untranslated words in the source intermediate linguistic structure.
18. The method of claim 1 wherein extracting features comprises:
calculating a perplexity of the target string with a statistical language model.