Using autoencoders for training natural language text classifiers
Systems and methods for using autoencoders for training natural language classifiers. An example method comprises: producing, by a computer system, a plurality of feature vectors, wherein each feature vector represents a natural language text of a text corpus, wherein the text corpus comprises a first plurality of annotated natural language texts and a second plurality of un-annotated natural language texts; training, using the plurality of feature vectors, an autoencoder represented by an artificial neural network; producing, by the autoencoder, an output of the hidden layer, by processing a training data set comprising the first plurality of annotated natural language texts; and training, using the training data set, a text classifier that accepts an input vector comprising the output of the hidden layer and yields a degree of association, with a certain text category, of a natural language text utilized to produce the output of the hidden layer.
1. A method, comprising:
producing, by a computer system, a plurality of feature vectors, wherein each feature vector represents a natural language text of a text corpus, wherein the text corpus comprises a first plurality of annotated natural language texts and a second plurality of un-annotated natural language texts;
training, using the plurality of feature vectors, an autoencoder represented by an artificial neural network, wherein the autoencoder comprises an input layer, a hidden layer, and an output layer;
producing, by the autoencoder, an output of the hidden layer, by processing a training data set comprising the first plurality of annotated natural language texts; and
training, using the training data set, a text classifier that accepts an input vector comprising the output of the hidden layer and yields a degree of association, with a certain text category, of a natural language text utilized to produce the output of the hidden layer.
2. The method of claim 1 , wherein a first dimension of the input layer is equal to a second dimension of the output layer and is greater than a third dimension of the hidden layer.
3. The method of claim 1 , wherein the hidden layer includes an activation function provided by a rectified linear unit.
4. The method of claim 1 , wherein training the autoencoder further comprisies:
determining a value of a weight parameter of the autoencoder to minimize an output error reflecting a difference between an input and output of the autoencoder.
5. The method of claim 1 , wherein training the autoencoder further comprisies:
determining a value of an activation function of the autoencoder to minimize an output error reflecting a difference between an input and output of the autoencoder.
6. The method of claim 1 , wherein training the autoencoder further comprisies:
determining a dimension of the hidden layer of the autoencoder to minimize an output error reflecting a difference between an input and output of the autoencoder.
7. The method of claim 1 , wherein training the text classifier further comprises:
determining a set of values of a plurality of parameters of the text classifier to optimize a fitness function representing a number of natural language texts of a validation data set that are correctly classified by the text classifier using the set of values.
8. The method of claim 1 , wherein each feature vector comprises a plurality of term frequency-inverse document frequency (TF-IDF) values, each value reflecting a frequency characteristic of a word identified by an index of the value in the feature vector.
9. The method of claim 1 , further comprising:
receiving a natural language text to be classified;
processing the natural language text by the autoencoder;
feeding an output of the hidden layer to the text classifier; and
determining, using the text classifier, a degree of association of the natural language text with a certain text category.
10. The method of claim 1 , further comprising:
utilizing the text classifier to perform a natural language processing task.
11. A system, comprising:
a memory;
a processor, coupled to the memory, the processor configured to:
produce a plurality of feature vectors, wherein each feature vector represents a natural language text of a text corpus, wherein the text corpus comprises a first plurality of annotated natural language texts and a second plurality of un-annotated natural language texts;
train, using the plurality of feature vectors, an autoencoder represented by an artificial neural network, wherein the autoencoder comprises an input layer, a hidden layer, and an output layer;
produce, by the autoencoder, an output of the hidden layer, by processing a training data set comprising the first plurality of annotated natural language texts; and
train, using the training data set, a text classifier that accepts an input vector comprising the output of the hidden layer and yields a degree of association, with a certain text category, of a natural language text utilized to produce the output of the hidden layer.
12. The system of claim 11 , wherein a first dimension of the input layer is equal to a second dimension of the output layer and is greater than a third dimension of the hidden layer.
13. The system of claim 11 , wherein each feature vector comprises a plurality of term frequency-inverse document frequency (TF-IDF) values, each value reflecting a frequency of occurrence, in the natural language text, of a word identified by an index of the value in the feature vector.
14. The system of claim 11 , wherein the processor is further configured to:
determine, using the text classifier, a degree of association of an input natural language text with a certain text category;
perform, based on the degree of association, a natural language processing task.
15. A non-transitory computer-readable storage medium comprising executable instructions that, when executed by a computer system, cause the computer system to:
produce a plurality of feature vectors, wherein each feature vector represents a natural language text of a text corpus, wherein the text corpus comprises a first plurality of annotated natural language texts and a second plurality of un-annotated natural language texts;
train, using the plurality of feature vectors, an autoencoder represented by an artificial neural network, wherein the autoencoder comprises an input layer, a hidden layer, and an output layer;
produce, by the autoencoder, an output of the hidden layer, by processing a training data set comprising the first plurality of annotated natural language texts; and
train, using the training data set, a text classifier that accepts an input vector comprising the output of the hidden layer and yields a degree of association, with a certain text category, of a natural language text utilized to produce the output of the hidden layer.
16. The non-transitory computer-readable storage medium of claim 15 , wherein a first dimension of the input layer is equal to a second dimension of the output layer and is greater than a third dimension of the hidden layer.
17. The non-transitory computer-readable storage medium of claim 15 , wherein training the text classifier further comprises:
determining a set of values of a plurality of parameters of the text classifier to optimize a fitness function representing a number of natural language texts of a validation data set that are correctly classified by the text classifier using the set of values.