Identifying codemixed text

1. A method for identifying codemixed text, the method comprising:

receiving, at data processing hardware, codemixed text;

segmenting, by the data processing hardware, the codemixed text into a plurality of tokens, each token comprising at least one character and delineated from any adjacent tokens by a space;

for each token of the codemixed text:

extracting, by the data processing hardware, features from the token; and

predicting, by the data processing hardware, a probability distribution over possible languages for the token using a language identifier model configured to receive the extracted features from the token as feature inputs;

assigning, by the data processing hardware, a language to each token of the codemixed text by executing a greedy search on the probability distribution over the possible languages predicted for each respective token; and

receiving, at the data processing hardware, an assignment constraint that assumes at least one language assigned to at least one token of the codemixed text,

wherein assigning the language to each token of the codemixed text comprises selecting the language having a greatest probability distribution for the respective token relative to any adjacent tokens based on the assignment constraint.

2. The method of claim 1 , further comprising, for each token of the codemixed text:

extracting, by the data processing hardware, features from any adjacent tokens,

wherein the language identifier model is further configured to receive the extracted features from the adjacent tokens as feature inputs for predicting the probability distribution over possible languages for the corresponding token.

3. The method of claim 1 , wherein the feature inputs comprise at least one of character features, script features, or lexicon features.

4. The method of claim 1 , wherein the at least one assumed language of the assignment constraint is English or French.

5. The method of claim 1 , wherein the assignment constraint comprises at least one fixed set of language pairs permissible for assignment to each token of the codemixed text.

6. A method for identifying codemixed text, the method comprising:

receiving, at data processing hardware, codemixed text;

segmenting, by the data processing hardware, the codemixed text into a plurality of tokens, each token comprising at least one character and delineated from any adjacent tokens by a space;

for each token of the codemixed text:

extracting, by the data processing hardware, features from the token by:

identifying all character n-gram features in the corresponding token; and

for each character n-gram feature, calculating a corresponding frequency of the character n-gram feature in the corresponding token by dividing a corresponding number of occurrences for the character n-gram in the corresponding token by a total number of character n-grams identified from the corresponding token; and

predicting, by the data processing hardware, a probability distribution over possible languages for the token using a language identifier model configured to receive the extracted features from the token as feature inputs; and

assigning, by the data processing hardware, a language to each token of the codemixed text by executing a greedy search on the probability distribution over the possible languages predicted for each respective token.

7. The method of claim 6 , wherein identifying all character n-gram features comprises identifying at least one of character unigram features, character bigram features, character trigram features, or character quadrigram features in the corresponding token.

8. A method for identifying codemixed text, the method comprising:

receiving, at data processing hardware, codemixed text;

segmenting, by the data processing hardware, the codemixed text into a plurality of tokens, each token comprising at least one character and delineated from any adjacent tokens by a space;

for each token of the codemixed text:

extracting, by the data processing hardware, features from the token by:

identifying all character script features in the corresponding token;

determining a unicode value for each identified character script feature; and

assigning each identified character script feature a corresponding character script type from a set of possible character script types based on the corresponding unicode value for the identified character script feature; and

predicting, by the data processing hardware, a probability distribution over possible languages for the token using a language identifier model configured to receive the extracted features from the token as feature inputs; and

assigning, by the data processing hardware, a language to each token of the codemixed text by executing a greedy search on the probability distribution over the possible languages predicted for each respective token.

9. The method of claim 8 , wherein the assigned character script type is associated with only one language.

10. A method for identifying codemixed text, the method comprising:

receiving, at data processing hardware, codemixed text;

segmenting, by the data processing hardware, the codemixed text into a plurality of tokens, each token comprising at least one character and delineated from any adjacent tokens by a space;

for each token of the codemixed text:

extracting, by the data processing hardware, features from the token by:

querying a lexicon library stored in memory hardware in communication with the data processing hardware, the lexicon library comprising a pool of word entries and corresponding language probability distributions for each word entry in the pool of word entries;

determining whether the token matches one of the word entries of the lexicon library; and

when the token matches one of the word entries of the lexicon library, retrieving the corresponding language probability distribution for the word entry that matches the token; and

predicting, by the data processing hardware, a probability distribution over possible languages for the token using a language identifier model configured to receive the extracted features from the token as feature inputs; and

assigning, by the data processing hardware, a language to each token of the codemixed text by executing a greedy search on the probability distribution over the possible languages predicted for each respective token.

11. A method for identifying codemixed text, the method comprising:

receiving, at data processing hardware, codemixed text;

segmenting, by the data processing hardware, the codemixed text into a plurality of tokens, each token comprising at least one character and delineated from any adjacent tokens by a space;

for each token of the codemixed text:

extracting, by the data processing hardware, features from the token; and

predicting, by the data processing hardware, a probability distribution over possible languages for the token using a language identifier model configured to receive the extracted features from the token as feature inputs; and

assigning, by the data processing hardware, a language to each token of the codemixed text by executing a greedy search on the probability distribution over the possible languages predicted for each respective token,

wherein the language identifier model includes a feed-forward neural network comprising:

an embedding layer configured to:

receive the feature inputs, each feature input comprising a sparse matrix;

map the sparse matrix of each respective feature input to dense embedding vectors, resulting in a learned embedding matrix; and

concatenate each learned embedding matrix corresponding to each received feature input;

a hidden layer interconnected with the embedding layer in a feed-forward manner, the hidden layer configured to:

receive the embedding layer; and

apply a rectified linear unit (ReLU) of the embedding layer; and

an output layer interconnected with the hidden layer in the feed-forward manner, the output layer configured to output a probability for each possible language for each respective token.

12. The method of claim 11 , wherein the output layer comprises a softmax layer.

13. A method for identifying codemixed text, the method comprising:

receiving, at data processing hardware, codemixed text;

segmenting, by the data processing hardware, the codemixed text into a plurality of tokens, each token comprising at least one character and delineated from any adjacent tokens by a space;

for each token of the codemixed text:

extracting, by the data processing hardware, features from the token; and

predicting, by the data processing hardware, a probability distribution over possible languages for the token using a language identifier model configured to receive the extracted features from the token as feature inputs, wherein the language identifier model executes a lexicon feature dropout strategy during training that drops a sub-set of extracted lexicon features as feature inputs; and

assigning, by the data processing hardware, a language to each token of the codemixed text by executing a greedy search on the probability distribution over the possible languages predicted for each respective token.

14. A system for identifying codemixed text, the system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations comprising:

receiving codemixed text;

segmenting the codemixed text into a plurality of tokens, each token comprising at least one character and delineated from any adjacent tokens by a space;

for each token of the codemixed text:

extracting features from the token; and

predicting a probability distribution over possible languages for the token using a language identifier model configured to receive the extracted features from the token as feature inputs;

assigning a language to each token of the codemixed text by executing a greedy search on the probability distribution over the possible languages predicted for each respective token; and

receiving an assignment constraint that assumes at least one language assigned to at least one token of the codemixed text,

wherein assigning the language to each token of the codemixed text comprises selecting the language having a greatest probability distribution for the respective token relative to any adjacent tokens based on the assignment constraint.

15. The system of claim 14 , wherein the operations further comprise, for each token of the codemixed text:

extracting features from any adjacent tokens,

wherein the language identifier model is further configured to receive the extracted features from the adjacent tokens as feature inputs for predicting the probability distribution over possible languages for the corresponding token.

16. The system of claim 14 , wherein the feature inputs comprise at least one of character features, script features, or lexicon features.

17. The system of claim 14 , wherein the at least one assumed language of the assignment constraint is English or French.

18. The system of claim 14 , wherein the assignment constraint comprises at least one fixed set of language pairs permissible for assignment to each token of the codemixed text.

19. A system for identifying codemixed text, the system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations comprising:

receiving codemixed text;

segmenting the codemixed text into a plurality of tokens, each token comprising at least one character and delineated from any adjacent tokens by a space;

for each token of the codemixed text:

extracting features from the token by:

identifying all character n-gram features in the corresponding token; and

for each character n-gram feature, calculating a corresponding frequency of the character n-gram feature in the corresponding token by dividing a corresponding number of occurrences for the character n-gram in the corresponding token by a total number of character n-grams identified from the corresponding token; and

predicting a probability distribution over possible languages for the token using a language identifier model configured to receive the extracted features from the token as feature inputs; and

assigning a language to each token of the codemixed text by executing a greedy search on the probability distribution over the possible languages predicted for each respective token.

20. The system of claim 19 , wherein identifying all character n-gram features comprises identifying at least one of character unigram features, character bigram features, character trigram features, or character quadrigram features in the corresponding token.

21. A system for identifying codemixed text, the system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations comprising:

receiving codemixed text;

segmenting the codemixed text into a plurality of tokens, each token comprising at least one character and delineated from any adjacent tokens by a space;

for each token of the codemixed text:

extracting features from the token by:

identifying all character script features in the corresponding token;

determining a unicode value for each identified character script feature; and

assigning each identified character script feature a corresponding character script type from a set of possible character script types based on the corresponding unicode value for the identified character script feature; and

predicting a probability distribution over possible languages for the token using a language identifier model configured to receive the extracted features from the token as feature inputs; and

assigning a language to each token of the codemixed text by executing a greedy search on the probability distribution over the possible languages predicted for each respective token.

22. The system of claim 21 , wherein the assigned character script type is associated with only one language.

23. A system for identifying codemixed text, the system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations comprising:

receiving codemixed text;

segmenting the codemixed text into a plurality of tokens, each token comprising at least one character and delineated from any adjacent tokens by a space;

for each token of the codemixed text:

extracting features from the token by:

querying a lexicon library stored in memory hardware in communication with the data processing hardware, the lexicon library comprising a pool of word entries and corresponding language probability distributions for each word entry in the pool of word entries;

determining whether the token matches one of the word entries of the lexicon library; and

when the token matches one of the word entries of the lexicon library, retrieving the corresponding language probability distribution for the word entry that matches the token; and

predicting a probability distribution over possible languages for the token using a language identifier model configured to receive the extracted features from the token as feature inputs; and

assigning a language to each token of the codemixed text by executing a greedy search on the probability distribution over the possible languages predicted for each respective token.

24. A system for identifying codemixed text, the system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations comprising:

receiving codemixed text;

segmenting the codemixed text into a plurality of tokens, each token comprising at least one character and delineated from any adjacent tokens by a space;

for each token of the codemixed text:

extracting features from the token; and

predicting a probability distribution over possible languages for the token using a language identifier model configured to receive the extracted features from the token as feature inputs; and

assigning a language to each token of the codemixed text by executing a greedy search on the probability distribution over the possible languages predicted for each respective token,

wherein the language identifier model includes a feed-forward neural network comprising:

an embedding layer configured to:

receive the feature inputs, each feature input comprising a sparse matrix;

map the sparse matrix of each respective feature input to dense embedding vectors, resulting in a learned embedding matrix; and

concatenate each learned embedding matrix corresponding to each received feature input;

a hidden layer interconnected with the embedding layer in a feed-forward manner, the hidden layer configured to:

receive the embedding layer; and

apply a rectified linear unit (ReLU) of the embedding layer; and

an output layer interconnected with the hidden layer in the feed-forward manner, the output layer configured to output a probability for each possible language for each respective token.

25. The system of claim 24 , wherein the output layer comprises a softmax layer.

26. A system for identifying codemixed text, the system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations comprising:

receiving codemixed text;

segmenting the codemixed text into a plurality of tokens, each token comprising at least one character and delineated from any adjacent tokens by a space;

for each token of the codemixed text:

extracting features from the token; and

predicting a probability distribution over possible languages for the token using a language identifier model configured to receive the extracted features from the token as feature inputs, wherein the language identifier model executes a lexicon feature dropout strategy during training that drops a sub-set of extracted lexicon features as feature inputs; and

assigning a language to each token of the codemixed text by executing a greedy search on the probability distribution over the possible languages predicted for each respective token.