System and Method for Language Identification
A system and method for training a language classifier are disclosed that may include obtaining an initial dictionary-based classifier model, stored in a computer memory, the model including a plurality of classifier n-grams; pruning away selected ones of the n-grams that do not significantly affect a performance of the classifier model; adding, to the model, selected supplemental n-grams that increase the effectiveness of the classifier model at identifying a language of a text sample, thereby growing the classifier model; and enabling the adding step to include adding n-grams of varying order, thereby enabling the provision of a variable-order model.
1 . A machine-implemented method for training a language classifier, the method comprising the steps of:
obtaining an initial dictionary-based classifier model, stored in a computer memory, the model including a plurality of classifier n-grams;
pruning away selected ones of the n-grams that do not significantly affect a performance of the classifier model;
adding, to the model, selected supplemental n-grams that increase the effectiveness of the classifier model at identifying a language of a text sample, thereby growing the classifier model; and
enabling the adding step to include adding n-grams of varying order, thereby enabling the provision of a variable-order model.
2 . The method of claim 1 further comprising the step of:
training the classifier model with interpolated modified Kneser-Ney smoothing.
3 . The method of claim 1 further comprising the step of:
modeling only a subset of the n-grams prior to the pruning step.
4 . The method of claim 1 wherein the adding step comprises:
using Kneser-Ney growing.
5 . The method of claim 1 wherein the pruning step comprises:
using Kneser pruning.
6 . The method of claim 1 further comprising the step of:
establishing a maximum order of the n-grams at a fixed value.
7 . The method of claim 1 further comprising the step of:
repeating the pruning and adding steps.
8 . A machine-implemented language identification method comprising:
storing variable-order n-gram language classifiers for a plurality of languages in a computer memory, thereby providing a plurality of respective language classifiers;
comparing a text message to each the plurality of classifiers using a processor;
determining a match probability score for each of the comparisons; and
identifying the language associated with the classifier incurring the highest match probability score as the language of the text message.
9 . The method of claim 8 wherein the variable-order n-grams correspond to one of the group consisting of: a variable number of letters; a variable number of phonemes; and a variable number of words.