Systems and methods for language detection
Implementations of the present disclosure are directed to a method, a system, and a computer program storage device for detecting a language in a text message. A plurality of different language detection tests are performed on a message associated with a user. Each language detection test determines a set of scores representing a likelihood that the message is in one of a plurality of different languages. One or more combinations of the score sets are provided as input to one or more distinct classifiers. Output from each of the classifiers includes a respective indication that the message is in one of the different languages. The language in the message may be identified as being the indicated language from one of the classifiers, based on a confidence score and/or an identified linguistic domain.
1. A computer-implemented method of identifying a language of a message, the method comprising:
performing a plurality of language detection tests on text, each language detection test determining a respective set of scores, each score in the set of scores representing a likelihood that the message is in a respective language of a plurality of different languages;
providing one or more combinations of the score sets as input to one or more distinct classifiers including a first classifier and a second classifier, wherein the first classifier was trained using outputs from a first combination of the language detection tests and the second classifier was trained using outputs from a different second combination of the language detection tests;
obtaining as output from each of the one or more classifiers a respective indication that the message is in one of the plurality of different languages, the indication comprising a confidence score; and
identifying the language of the message based on one of the confidence scores.
2. The method of claim 1 wherein a particular output is respective scores each representing a likelihood that a respective message is in one of a plurality of different languages.
3. The method of claim 1 , wherein a particular classifier is a supervised learning model, a partially supervised learning model, an unsupervised learning model, or an interpolation.
4. The method of claim 1 , wherein identifying the language of the message comprises selecting the confidence score based on an expected language detection accuracy.
5. The method of claim 1 , wherein identifying the language of the message comprises selecting the confidence score based on the linguistic domain of the message.
6. The method of claim 1 , wherein the message comprises two or more of the following: a letter, a number, a symbol, and an emoticon.
7. The method of claim 1 , wherein a particular language detection test is a byte n-gram method, a dictionary-based method, an alphabet-based method, or a script-based method.
8. The method of claim 1 , wherein the one or more combinations comprise score sets from a byte n-gram method and a dictionary-based method.
9. The method of claim 1 , wherein the one or more combinations further comprise score sets from at least one of a script-based method and an alphabet-based method.
10. A system comprising:
one or more computers programmed to perform operations comprising:
performing a plurality of language detection tests on text, each language detection test determining a respective set of scores, each score in the set of scores representing a likelihood that the message is in a respective language of a plurality of different languages;
providing one or more combinations of the score sets as input to one or more distinct classifiers including a first classifier and a second classifier, wherein the first classifier was trained using outputs from a first combination of the language detection tests and the second classifier was trained using outputs from a different second combination of the language detection tests;
obtaining as output from each of the one or more classifiers a respective indication that the message is in one of the plurality of different languages, the indication comprising a confidence score; and
identifying the language of the message based on one of the confidence scores.
11. The system of claim 10 wherein a particular output is respective scores each representing a likelihood that a respective message is in one of a plurality of different languages.
12. The system of claim 10 , wherein a particular classifier is a supervised learning model, a partially supervised learning model, an unsupervised learning model, or an interpolation.
13. The system of claim 10 , wherein identifying the language of the message comprises selecting the confidence score based on an expected language detection accuracy.
14. The system of claim 10 , wherein identifying the language of the message comprises selecting the confidence score based on the linguistic domain of the message.
15. The system of claim 10 , wherein the message comprises two or more of the following: a letter, a number, a symbol, and an emoticon.
16. The system of claim 10 , wherein a particular language detection test is a byte n-gram system, a dictionary-based system, an alphabet-based system, or a script-based system.
17. The system of claim 10 , wherein the one or more combinations comprise score sets from a byte n-gram system and a dictionary-based system.
18. The system of claim 10 , wherein the one or more combinations further comprise score sets from at least one of a script-based system and an alphabet-based system.
19. An article comprising a non-transitory computer-readable medium having instructions stored thereon that, when executed by a computer, perform operations comprising:
performing a plurality of language detection tests on text, each language detection test determining a respective set of scores, each score in the set of scores representing a likelihood that the message is in a respective language of a plurality of different languages;
providing one or more combinations of the score sets as input to one or more distinct classifiers including a first classifier and a second classifier, wherein the first classifier was trained using outputs from a first combination of the language detection tests and the second classifier was trained using outputs from a different second combination of the language detection tests;
obtaining as output from each of the one or more classifiers a respective indication that the message is in one of the plurality of different languages, the indication comprising a confidence score; and
identifying the language of the message based on one of the confidence scores.
20. The article of claim 19 wherein a particular output is respective scores each representing a likelihood that a respective message is in one of a plurality of different languages.
21. The article of claim 19 , wherein a particular classifier is a supervised learning model, a partially supervised learning model, an unsupervised learning model, or an interpolation.
22. The article of claim 19 , wherein identifying the language of the message comprises selecting the confidence score based on an expected language detection accuracy.
23. The article of claim 19 , wherein identifying the language of the message comprises selecting the confidence score based on the linguistic domain of the message.
24. The article of claim 19 , wherein the message comprises two or more of the following: a letter, a number, a symbol, and an emoticon.
25. The article of claim 19 , wherein a particular language detection test is a byte n-gram system, a dictionary-based system, an alphabet-based system, or a script-based system.
26. The article of claim 19 , wherein the one or more combinations comprise score sets from a byte n-gram system and a dictionary-based system.
27. The article of claim 19 , wherein the one or more combinations further comprise score sets from at least one of a script-based system and an alphabet-based system.