Systems and methods for language detection

1. A computer-implemented method of identifying a language in a message, the method comprising:

obtaining a text message generated by a user;

removing non-language characters from the text message to generate a sanitized text message;

detecting an alphabet and a script present in the sanitized text message, wherein

(i) detecting the alphabet comprises performing an alphabet-based language detection test to determine a first set of scores, and wherein each score in the first set of scores represents a likelihood that the sanitized text message comprises the alphabet for one of a plurality of different languages, and

(ii) detecting the script comprises performing a script-based language detection test to determine a second set of scores, and wherein each score in the second set of scores represents a likelihood that the sanitized text message comprises the script for one of the plurality of different languages;

providing one or more combinations of the first and second sets of scores as input to one or more classifiers including a first classifier and a second classifier, wherein the first classifier was trained using outputs from a first combination of language detection tests and the second classifier was trained using outputs from a second combination of language detection tests;

obtaining as output from at least one of the one or more classifiers a respective confidence score that the sanitized text message is in one of a plurality of different languages; and

identifying the language in the sanitized text message based on the confidence score from at least one of the one or more classifiers.

2. The method of claim 1 , wherein the non-language characters comprise at least one of an emoji, a punctuation mark, an extra space, a carriage return, and a numerical character.

3. The method of claim 1 , wherein the one or more combinations comprise an interpolation between the first and second sets of scores.

4. The method of claim 1 , wherein providing one or more combinations of the first and second sets of scores comprises:

performing a language detection test on the sanitized text message to generate a third set of scores, wherein each score in the third set of scores represents a likelihood that the sanitized text message comprises one of a plurality of different languages.

5. The method of claim 4 , wherein the language detection test is selected from a plurality of language detection tests based on at least one of the first set of scores, the second set of scores, and a combination of the first and second sets of scores.

6. The method of claim 4 , wherein the language detection test comprises a language detection method.

7. The method of claim 6 , wherein the language detection method comprises at least one of a dictionary-based language detection test, an n-gram language detection test, and a user language profile language detection test.

8. The method of claim 4 , comprising:

processing the third set of scores using the one or more classifiers to identify the language in the sanitized text message.

9. The method of claim 1 , wherein the one or more classifiers comprise at least one of a supervised learning model, a partially supervised learning model, an unsupervised learning model, and an interpolation.

10. A computer-implemented system for identifying a language in a message, comprising:

one or more computer processors programmed to implement a sanitizer module, a grouper module, and a language detector module,

wherein the sanitizer module obtains a text message generated by a user and removes non-language characters from the text message to generate a sanitized text message,

wherein the grouper module detects an alphabet and a script present in the sanitized text message, and wherein

(i) detecting the alphabet comprises performing an alphabet-based language detection test to determine a first set of scores, and wherein each score in the first set of scores represents a likelihood that the sanitized text message comprises the alphabet for one of a plurality of different languages, and

(ii) detecting the script comprises performing a script-based language detection test to determine a second set of scores, and wherein each score in the second set of scores represents a likelihood that the sanitized text message comprises the script for one of the plurality of different languages, and

wherein the language detector module is operable to perform operations comprising:

providing one or more combinations of the first and second sets of scores as input to one or more classifiers including a first classifier and a second classifier, wherein the first classifier was trained using outputs from a first combination of language detection tests and the second classifier was trained using outputs from a second combination of language detection tests;

obtaining as output from at least one of the one or more classifiers a respective confidence score that the sanitized text message is in one of a plurality of different languages; and

identifying the language in the sanitized text message based on the confidence score from at least one of the one or more classifiers.

11. The system of claim 10 , wherein the non-language characters comprise at least one of an emoji, a punctuation mark, an extra space, a carriage return, and a numerical character.

12. The system of claim 10 , wherein the one or more combinations comprise an interpolation between the first and second sets of scores.

13. The system of claim 10 , wherein the grouper module is operable to perform operations comprising:

selecting the language detector module from a plurality of language detector modules based on at least one of the first set of scores, the second set of scores, and a combination of the first and second sets of scores.

14. The system of claim 10 , wherein the language detector module comprises:

a language detection methods module operable to perform operations comprising:

performing a language detection test on the sanitized text message to generate a third set of scores, wherein each score in the third set of scores represents a likelihood that the sanitized text message comprises one of a plurality of different languages.

15. The system of claim 14 , wherein the language detection test comprises at least one of a dictionary-based language detection test, an n-gram language detection test, and a user language profile language detection test.

16. The system of claim 14 , wherein the language detector module comprises:

a classifier module operable to perform operations comprising:

processing the third set of scores using the one or more classifiers to identify the language in the sanitized text message.

17. The system of claim 16 , wherein the classifier module is operable to perform operations comprising:

outputting an indication that the sanitized text message is in the identified language, wherein the indication comprises a confidence score.

18. The system of claim 10 , wherein the one or more classifiers comprise at least one of a supervised learning model, a partially supervised learning model, an unsupervised learning model, and an interpolation.

19. An article, comprising:

a non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more computers, cause the computers to perform operations comprising:

obtaining a text message generated by a user;

removing non-language characters from the text message to generate a sanitized text message;

detecting an alphabet and a script present in the sanitized text message, wherein

(i) detecting the alphabet comprises performing an alphabet-based language detection test to determine a first set of scores, and wherein each score in the first set of scores represents a likelihood that the sanitized text message comprises the alphabet for one of a plurality of different languages, and

(ii) detecting the script comprises performing a script-based language detection test to determine a second set of scores, and wherein each score in the second set of scores represents a likelihood that the sanitized text message comprises the script for one of the plurality of different languages;

providing one or more combinations of the first and second sets of scores as input to one or more classifiers including a first classifier and a second classifier, wherein the first classifier was trained using outputs from a first combination of language detection tests and the second classifier was trained using outputs from a second combination of language detection tests;

obtaining as output from at least one of the one or more classifiers a respective confidence score that the sanitized text message is in one of a plurality of different languages; and

identifying the language in the sanitized text message based on the confidence score from at least one of the one or more classifiers.