Machine learned language modeling and identification
Systems, devices, media, and methods are presented for generating a language detection model of a language analysis system. The systems and methods access a set of messages including text elements and convert the set of messages into a set of training messages. The set of training messages are configured for training a language detection model. The systems and methods train a classifier based on the set of training messages. The classifier has a set of features representing word frequency, character frequency, and a character ratio. The systems and methods generate a language detection model based on the classifier and the set of features.
1. A method comprising:
accessing a set of messages comprising text and graphical text elements, the graphical text elements conveying an emotion;
converting the set of messages into a set of training messages, the set of training messages configured for training a language detection machine-learning model;
training a classifier based on the set of training messages, the classifier having a set of features representing term frequency inverse document frequency (TF-IDF) values of words, TF-IDF values of characters, a message locale and a character ratio, wherein the character ratio is based on a number of non-American Standard Code for Information Interchange (ASCII) characters included in a message; and
generating a language detection model based on the classifier and the set of features.
2. The method of claim 1 , wherein converting the set of messages into the set of training messages further comprises:
identifying a subset of messages having a number of characters below a character threshold; and
removing the subset of messages from the set of messages.
3. The method of claim 1 , wherein converting the set of messages into the set of training messages further comprises:
identifying a subset of messages containing text representing a named entity;
for each message, determining a number of characters of the message, in addition to the text representing the named entity, is above a character threshold; and
removing the text representing the named entity from the subset of messages.
4. The method of claim 1 , wherein converting the set of messages into the set of training messages further comprises:
identifying a subset of messages having text with repeating characters;
for each message, determining the repeating characters include one or more superfluous characters; and
for each message, normalizing the one or more superfluous characters.
5. The method of claim 1 , wherein the TF-IDF values of words represents word frequencies for a plurality of word n-grams.
6. The method of claim 1 , wherein the TF-IDF values of characters represents character frequencies for a plurality of character n-grams.
7. The method of claim 1 , wherein the character ratio represents a number of characters, excluded from a known character set, relative to a length of a message.
8. The method of claim 1 , wherein the message locale comprises information relating to a location of a user.
9. A system comprising:
one or more processors; and
a non-transitory processor-readable storage medium storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
accessing a set of messages comprising text and graphical text elements, the graphical text elements conveying an emotion;
converting the set of messages into a set of training messages, the set of training messages configured for training a language detection machine-learning model;
training a classifier based on the set of training messages, the classifier having a set of features representing term frequency inverse document frequency (TF-IDF) values of words, TF-IDF values of characters, a message locale and a character ratio, wherein the character ratio is based on a number of non-American Standard Code for Information Interchange (ASCII) characters included in a message; and
generating a language detection model based on the classifier and the set of features.
10. The system of claim 9 , wherein converting the set of messages into the set of training messages further comprises:
identifying a subset of messages having a number of characters below a character threshold; and
removing the subset of messages from the set of messages.
11. The system of claim 9 , wherein converting the set of messages into the set of training messages further comprises:
identifying a subset of messages containing text representing a named entity;
for each message, determining a number of characters of the message, in addition to the text representing the named entity, is above a character threshold; and
removing the text representing the named entity from the subset of messages.
12. The system of claim 9 , wherein converting the set of messages into the set of training messages further comprises:
identifying a subset of messages having text with repeating characters;
for each message, determining the repeating characters include one or more superfluous characters; and
for each message, normalizing the one or more superfluous characters.
13. The system of claim 9 , wherein the TF-IDF values of words represents word frequencies for a plurality of word n-grams and the TF-IDF values of characters represents character frequencies for a plurality of character n-grams.
14. The system of claim 9 , wherein the character ratio represents a number of characters, excluded from a known character set, relative to a length of a message.
15. A non-transitory processor-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:
accessing a set of messages comprising text and graphical text elements, the graphical text elements conveying an emotion;
converting the set of messages into a set of training messages, the set of training messages configured for training a language detection machine-learning model;
training a classifier based on the set of training messages, the classifier having a set of features representing term frequency inverse document frequency (TF-IDF) values of words, TF-IDF values of characters, a message locale and a character ratio, wherein the character ratio is based on a number of non-American Standard Code for Information Interchange (ASCII) characters included in a message; and
generating a language detection model based on the classifier and the set of features.
16. The non-transitory processor-readable storage medium of claim 15 , wherein converting the set of messages into the set of training messages further comprises:
identifying a subset of messages having a number of characters below a character threshold; and
removing the subset of messages from the set of messages.
17. The non-transitory processor-readable storage medium of claim 15 , wherein converting the set of messages into the set of training messages further comprises:
identifying a subset of messages containing text representing a named entity;
for each message, determining a number of characters of the message, in addition to the text representing the named entity, is above a character threshold; and
removing the text representing the named entity from the subset of messages.
18. The non-transitory processor-readable storage medium of claim 15 , wherein converting the set of messages into the set of training messages further comprises:
identifying a subset of messages having text with repeating characters;
for each message, determining the repeating characters include one or more superfluous characters; and
for each message, normalizing the one or more superfluous characters.
19. The non-transitory processor-readable storage medium of claim 15 , wherein the TF-IDF values of words represents word frequencies for a plurality of word n-grams and the TF-IDF values of characters represents character frequencies for a plurality of character n-grams.
20. The non-transitory processor-readable storage medium of claim 15 , the character ratio represents a number of characters, excluded from a known character set, relative to a length of a message.