Multi-term query subsumption for document classification
In general, embodiments of the present invention provide systems, methods and computer readable media for generating an optimal classifying query set for categorizing and/or labeling textual data based on a query subsumption calculus to determine, given two queries, whether one of the queries subsumes another. In one aspect, a method includes generating a group of determining queries based on analyzing text within a document; receiving a group of classifying queries; and, for each determining query within the group of determining queries, determining whether at least one of the classifying queries is subsumed by the determining query; and updating the group of classifying queries in an instance in which the classifying query is subsumed by the determining query.
1. A computer-implemented method, comprising:
generating a determining queries group based on analyzing document text within a document, wherein each determining query within the determining queries group includes at least one term identified within the document text;
receiving a classifying queries group, the classifying queries group representing a particular category;
updating the classifying queries group in an instance in which a classifying query is subsumed by a determining query, wherein subsuming is defined by the determining query and the classifying query sharing at least two common terms;
identifying a classifying queries subsuming subset, the classifying queries subsuming subset being a subset of the classifying queries group, wherein each classifying query of the classifying queries subsuming subset each subsumes at least one of the determining queries of the determining queries group;
calculating a document categorization score based on a normalized sum of respective performance metrics for respective classifying queries of the classifying queries subsuming subset; and
associating the document with the particular category represented by the classifying queries group in an instance in which the document categorization score is greater than a categorization threshold value.
2. The computer-implemented method of claim 1 , wherein the respective performance metrics are respective binormal separation scores for the respective classifying queries of the classifying queries subsuming subset.
3. The computer-implemented method of claim 2 , wherein the respective binormal separation scores are calculated based on training data used to generate the classifying queries group.
4. The computer-implemented method of claim 1 , wherein the categorization threshold value is computed through cross-validation at a time of training a machine learning model.
5. The computer-implemented method of claim 1 , wherein the document categorization threshold value represents a score threshold that yields a minimum desired performance metric.
6. The computer-implemented method of claim 1 , wherein the document categorization threshold value is optimized for an F-score.
7. The computer-implemented method of claim 2 , wherein the binormal separation score of a classifying query represents how well the classifying query separates positive documents from negative documents in a corpus.
8. The computer-implemented method of claim 7 , wherein the positive documents have been categorized as belonging to a category represented by a feature set and the negative documents have been categorized as belonging to a different category than the category represented by the feature set.
9. A system, comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
generating a determining queries group based on analyzing document text within a document, wherein each determining query within the determining queries group includes at least one term identified within the document text;
receiving a classifying queries group, the classifying queries group representing a particular category;
updating the classifying queries group in an instance in which a classifying query is subsumed by a determining query, wherein subsuming is defined by the determining query and the classifying query sharing at least two common terms;
identifying a classifying queries subsuming subset, the classifying queries subsuming subset being a subset of the classifying queries group, wherein each classifying query of the classifying queries subsuming subset each subsumes at least one of the determining queries of the determining queries group;
calculating a document categorization score based on a normalized sum of respective performance metrics for respective classifying queries of the classifying queries subsuming subset; and
associating the document with the particular category represented by the classifying queries group in an instance in which the document categorization score is greater than a categorization threshold value.
10. They system of claim 9 , wherein the respective performance metrics are respective binormal separation scores for the respective classifying queries of the classifying queries subsuming subset.
11. They system of claim 10 , wherein the respective binormal separation scores are calculated based on training data used to generate the classifying queries group.
12. They system of claim 9 , wherein the categorization threshold value is computed through cross-validation at a time of training a machine learning model.
13. They system of claim 9 , wherein the document categorization threshold value represents a score threshold that yields a minimum desired performance metric.
14. They system of claim 9 , wherein the document categorization threshold value is optimized for an F-score.
15. They system of claim 10 , wherein the binormal separation score of a classifying query represents how well the classifying query separates positive documents from negative documents in a corpus.
16. They system of claim 15 , wherein the positive documents have been categorized as belonging to a category represented by a feature set and the negative documents have been categorized as belonging to a different category than the category represented by the feature set.