Systems and methods for language detection
Implementations of the present disclosure are directed to a method, a system, and a computer program storage device for detecting a language in a text message. A plurality of different language detection tests are performed on a message associated with a user. Each language detection test determines a set of scores representing a likelihood that the message is in one of a plurality of different languages. One or more combinations of the score sets are provided as input to one or more distinct classifiers. Output from each of the classifiers includes a respective indication that the message is in one of the different languages. The language in the message may be identified as being the indicated language from one of the classifiers, based on a confidence score and/or an identified linguistic domain.
1. A computer-implemented method of identifying a language of a message, the method comprising:
training a first classifier using training data comprising collections of first score sets from different language detection tests and an indication of the correct language for each collection of score sets wherein each first score set comprises a plurality of respective scores each representing a likelihood that a respective first message is in one of a plurality of different languages;
performing a plurality of the language detection tests on text in a message authored by a user, each language detection test determining a respective set of scores, each score in the set of scores representing a likelihood that the message is in a respective language of the plurality of different languages;
providing one or more combinations of the score sets as input to one or more distinct classifiers including the first classifier;
obtaining as output from each of the one or more classifiers a respective indication that the message is in one of the plurality of different languages, the indication comprising a confidence score; and
identifying the language of the message based on one of the confidence scores.
2. The method of claim 1 , wherein a particular classifier is a supervised learning model, a partially supervised learning model, an unsupervised learning model, or an interpolation.
3. The method of claim 1 , wherein identifying the language of the message comprises selecting the confidence score based on an expected language detection accuracy.
4. The method of claim 1 , wherein identifying the language of the message comprises selecting the confidence score based on the linguistic domain of the message.
5. The method of claim 1 , wherein the message comprises two or more of the following: a letter, a number, a symbol, and an emoticon.
6. The method of claim 1 , wherein the plurality of different language detection tests comprises at least two methods selected from the group consisting of a byte n-gram method, a dictionary-based method, an alphabet-based method, and a script-based method.
7. The method of claim 1 , wherein the plurality of different language detection tests are performed simultaneously.
8. The method of claim 1 , wherein the one or more combinations comprise score sets from a byte n-gram method and a dictionary-based method.
9. The method of claim 8 , wherein the one or more combinations further comprise score sets from at least one of a script-based method and an alphabet-based method.
10. The method of claim 1 , wherein the score sets comprise at least one score from a user language profile method that identifies a language preference for the user based on previous messages authored by the user.
11. A system for identifying a language of a message, the system comprising:
a computer storage device having instructions stored thereon; and
a data processing apparatus configured to execute the instructions to perform operations comprising:
training a first classifier using training data comprising collections of first score sets from different language detection tests and an indication of the correct language for each collection of score sets wherein each first score set comprises a plurality of respective scores each representing a likelihood that a respective first message is in one of a plurality of different languages;
performing a plurality of the language detection tests on text in a message authored by a user, each language detection test determining a respective set of scores, each score in the set of scores representing a likelihood that the message is in a respective language of the plurality of different languages;
providing one or more combinations of the score sets as input to one or more distinct classifiers including the classifier;
obtaining as output from each of the one or more classifiers a respective indication that the message is in one of the plurality of different languages, the indication comprising a confidence score; and
identifying the language of the message based on one of the confidence scores.
12. The system of claim 11 , wherein a particular classifier is a supervised learning model, a partially supervised learning model, an unsupervised learning model, or an interpolation.
13. The system of claim 11 , wherein identifying the language of the message comprises selecting the confidence score based on an expected language detection accuracy.
14. The system of claim 11 , wherein identifying the language of the message comprises selecting the confidence score based on the linguistic domain of the message.
15. The system of claim 11 , wherein the message comprises two or more of the following: a letter, a number, a symbol, and an emoticon.
16. The system of claim 11 , wherein the plurality of different language detection tests comprises at least two methods selected from the group consisting of a byte n-gram method, a dictionary-based method, an alphabet-based method, and a script-based method.
17. The system of claim 11 , wherein the plurality of different language detection tests are performed simultaneously.
18. The system of claim 11 , wherein the one or more combinations comprise score sets from a byte n-gram method and a dictionary-based method.
19. The system of claim 18 , wherein the one or more combinations further comprise score sets from at least one of a script-based method and an alphabet-based method.
20. The system of claim 11 , wherein the score sets comprise at least one score from a user language profile method that identifies a language preference for the user based on previous messages authored by the user.
21. A computer program product stored in one or more storage devices for controlling a processing mode of a data processing apparatus, the computer program product being executable by the data processing apparatus to cause the data processing apparatus to perform operations comprising:
training a first classifier using training data comprising collections of first score sets from different language detection tests and an indication of the correct language for each collection of score sets wherein each first score set comprises a plurality of respective scores each representing a likelihood that a respective first message is in one of a plurality of different languages;
performing a plurality of the language detection tests on text in a message authored by a user, each language detection test determining a respective set of scores, each score in the set of scores representing a likelihood that the message is in a respective language of the plurality of different languages;
providing one or more combinations of the score sets as input to one or more distinct classifiers including the first classifier;
obtaining as output from each of the one or more classifiers a respective indication that the message is in one of the plurality of different languages, the indication comprising a confidence score; and
identifying the language of the message based on one of the confidence scores.
22. The computer program product of claim 21 , wherein a particular classifier is a supervised learning model, a partially supervised learning model, an unsupervised learning model, or an interpolation.
23. The computer program product of claim 21 , wherein identifying the language of the message comprises selecting the confidence score based on an expected language detection accuracy.
24. The computer program product of claim 21 , wherein identifying the language of the message comprises selecting the confidence score based on the linguistic domain of the message.
25. The computer program product of claim 21 , wherein the message comprises two or more of the following: a letter, a number, a symbol, and an emoticon.
26. The computer program product of claim 21 , wherein the plurality of different language detection tests comprises at least two methods selected from the group consisting of a byte n-gram method, a dictionary-based method, an alphabet-based method, and a script-based method.
27. The computer program product of claim 21 , wherein the plurality of different language detection tests are performed simultaneously.
28. The computer program product of claim 21 , wherein the one or more combinations comprise score sets from a byte n-gram method and a dictionary-based method.
29. The computer program product of claim 28 , wherein the one or more combinations further comprise score sets from at least one of a script-based method and an alphabet-based method.
30. The computer program product of claim 21 , wherein the score sets comprise at least one score from a user language profile method that identifies a language preference for the user based on previous messages authored by the user.