Using autoencoders for training natural language text classifiers
Systems and methods for using autoencoders for training natural language classifiers. An example method comprises: producing, by a computer system, a plurality of feature vectors, wherein each feature vector represents a natural language text of a text corpus, wherein the text corpus comprises a first plurality of annotated natural language texts and a second plurality of un-annotated natural language texts; training, using the plurality of feature vectors, an autoencoder represented by an artificial neural network; producing, by the autoencoder, an output of the hidden layer, by processing a training data set comprising the first plurality of annotated natural language texts; and training, using the training data set, a text classifier that accepts an input vector comprising the output of the hidden layer and yields a degree of association, with a certain text category, of a natural language text utilized to produce the output of the hidden layer.
1. A method, comprising:
receiving, by a computer system, a natural language text;
processing the natural language text by an autoencoder represented by an artificial neural network comprising an input layer, a hidden layer, and an output layer;
feeding, to a text classifier, an input vector comprising an output of the hidden layer; and
determining, using the text classifier, a degree of association of the natural language text with a certain text category.
2. The method of claim 1 , wherein the hidden layer comprises an activation function provided by a rectified linear unit.
3. The method of claim 1 , wherein a first dimension of the input layer is equal to a second dimension of the output layer and is greater than a third dimension of the hidden layer.
4. The method of claim 1 , further comprising:
producing a feature vector comprising a plurality of term frequency-inverse document frequency (TF-IDF) values, each value reflecting a frequency of occurrence, in the natural language text, of a word identified by an index of the value in the feature vector; and
producing the input vector by concatenating the output of the hidden layer and the feature vector.
5. The method of claim 1 , further comprising:
producing a plurality of classifier input vectors, wherein each classifier input vector comprises an output of the hidden layer; and
training the text classifier using the plurality of classifier input vectors, wherein each classifier input vector is associated with a known category of a training natural language text that has been utilized for producing the output of the hidden layer.
6. The method of claim 1 , further comprising:
producing a plurality of classifier input vectors, wherein each classifier input vector comprises a combination of a feature vector representing a natural language text and an output of the hidden layer produced by processing the natural language text; and
training the text classifier using the plurality of classifier input vectors, wherein each classifier input vector is associated with a known category of a training natural language text that has been utilized for producing the output of the hidden layer.
7. The method of claim 1 , further comprising:
producing a plurality of feature vectors, wherein each feature vector represents a training natural language text of a text corpus, wherein the text corpus comprises a first plurality of annotated natural language texts and a second plurality of un-annotated natural language texts;
training the autoencoder using the plurality of feature vectors.
8. The method of claim 1 , further comprising:
performing, based on the degree of association, a natural language processing task.
9. A system, comprising:
a memory;
a processor, coupled to the memory, the processor configured to:
receive a natural language text;
process the natural language text by an autoencoder represented by an artificial neural network comprising an input layer, a hidden layer, and an output layer;
feed, to a text classifier, an input vector comprising an output of the hidden layer; and
determine, using the text classifier, a degree of association of the natural language text with a certain text category.
10. The system of claim 9 , wherein the hidden layer comprises an activation function provided by a rectified linear unit.
11. The system of claim 9 , wherein a first dimension of the input layer is equal to a second dimension of the output layer and is greater than a third dimension of the hidden layer.
12. The system of claim 9 , wherein the processor is further configured to:
produce a feature vector comprising a plurality of term frequency-inverse document frequency (TF-IDF) values, each value reflecting a frequency of occurrence, in the natural language text, of a word identified by an index of the value in the feature vector; and
producing the input vector by concatenating the output of the hidden layer and the feature vector.
13. The system of claim 9 , wherein the processor is further configured to:
produce a plurality of classifier input vectors, wherein each classifier input vector comprises an output of the hidden layer; and
train the text classifier using the plurality of classifier input vectors, wherein each classifier input vector is associated with a known category of a training natural language text that has been utilized for producing the output of the hidden layer.
14. The system of claim 9 , wherein the processor is further configured to:
produce a plurality of classifier input vectors, wherein each classifier input vector comprises a combination of a feature vector representing a natural language text and an output of the hidden layer produced by processing the natural language text; and
training the text classifier using the plurality of classifier input vectors, wherein each classifier input vector is associated with a known category of a training natural language text that has been utilized for producing the output of the hidden layer.
15. The system of claim 9 , wherein the processor is further configured to:
produce a plurality of feature vectors, wherein each feature vector represents a training natural language text of a text corpus, wherein the text corpus comprises a first plurality of annotated natural language texts and a second plurality of un-annotated natural language texts;
training the autoencoder using the plurality of feature vectors.
16. The system of claim 9 , wherein the processor is further configured to:
perform, based on the degree of association, a natural language processing task.
17. A non-transitory computer-readable storage medium comprising executable instructions that, when executed by a computer system, cause the computer system to:
receive a natural language text;
process the natural language text by an autoencoder represented by an artificial neural network comprising an input layer, a hidden layer, and an output layer;
feed, to a text classifier, an input vector comprising an output of the hidden layer; and
determine, using the text classifier, a degree of association of the natural language text with a certain text category.
18. The non-transitory computer-readable storage medium of claim 17 , further comprising executable instructions that, when executed by the computer system, cause the computer system to:
perform, based on the degree of association, a natural language processing task.
19. The non-transitory computer-readable storage medium of claim 17 , wherein a first dimension of the input layer is equal to a second dimension of the output layer and is greater than a third dimension of the hidden layer.
20. The non-transitory computer-readable storage medium of claim 17 , further comprising executable instructions that, when executed by the computer system, cause the computer system to:
produce a feature vector comprising a plurality of term frequency-inverse document frequency (TF-IDF) values, each value reflecting a frequency of occurrence, in the natural language text, of a word identified by an index of the value in the feature vector; and
producing the input vector by concatenating the output of the hidden layer and the feature vector.