Discriminative training of language models for text and speech classification
View Patent ↗Methods are disclosed for estimating language models such that the conditional likelihood of a class given a word string, which is very well correlated with classification accuracy, is maximized. The methods comprise tuning statistical language model parameters jointly for all classes such that a classifier discriminates between the correct class and the incorrect ones for a given training sentence or utterance. Specific embodiments of the present invention pertain to implementation of the rational function growth transform in the context of a discriminative training technique for n-gram classifiers.
1. A computer implemented method of classifying a natural language input, comprising:
training a plurality of statistical classification components jointly in relation to one another to maximize a conditional likelihood of a class given a word string using an application of the rational growth transform, the plurality of statistical classification components being n-gram language model classifiers that each correspond to a different class, wherein each class corresponds to a different category of subject matter, and wherein training the plurality of statistical classification components comprises:
identifying an optimal number of rational function growth transform iterations and an optimal conditional maximum likelihood (CML) weight β max to facilitate application of the rational function growth transform, wherein identifying comprises:
splitting a collection of training data into a collection of main data and a collection of held-out data;
using the main data to estimate a series of relative frequencies for the statistical classification components; and
using the held-out data to tune the optimal number of rational function growth transform iterations and the optimal CML weight β max ;
receiving a natural language input;
applying the plurality of statistical classification components to the natural language input so as to classify the natural language input into a particular one or more of the plurality of classes that represent the category or categories of subject matter that is best correlated to the natural language input;
wherein using the held-out data to tune comprises:
fixing a preset number N of rational function growth transform iterations to be run;
fixing a range of values to be explored for determining the optimal CML weight β max ;
for each value β max , running as many rational function growth transform functions as possible up to N such that the conditional likelihood of the main data increases at each iteration; and
identifying as optimal the number of rational function growth transforms iterations; and the β max value that maximizes the conditional likelihood of the held-out data; and
wherein training the plurality of statistical classification components further comprises:
pooling the main and held-out data to form a combined collection of training data; and
training the plurality of statistical classification components on the combined collection of training data using the optimal number of rational function growth transform iterations and the optimal CML weight β max .
2. The method of claim 1 , wherein the plurality of statistical classification components are n-gram language models.
3. The method of claim 1 , wherein the natural language input is text that is generated by a speech recognition engine.
4. The method of claim 1 , wherein the natural language input is a textual representation of a recognized utterance.
5. The method of claim 1 , wherein the natural language input is text generated as a textual representation of a handwritten input.
6. The method of claim 1 , further comprising preprocessing the natural language input.
7. The method of claim 6 , wherein preprocessing comprises removing an article of speech.
8. The method of claim 6 , wherein preprocessing comprises transforming the natural language input into an indirect representation.
9. The method of claim 8 , wherein transforming comprises transforming into a vector representation.
10. The method of claim 6 , wherein preprocessing comprises reducing the natural language input to a select set of extracted features.
11. The method of claim 6 , wherein preprocessing comprises reducing the natural language input to only include words included with a predefined vocabulary.
12. The method of claim 6 , wherein preprocessing comprises adding an indication of the presence of absence of words previously determined to carry a predefined type of content.
13. The method of claim 6 , wherein preprocessing comprises stemming.