Identifying and processing a number of features identified in a document to determine a type of the document
A system and method for document classification are presented. An input document is received (e.g., by at least one server communicatively coupled to a network). A plurality of features are identified in the input document. The plurality of features include sequences of text extracted from the input document. A feature vector of the input document is generated based upon the sequences of text, and the feature vector of the input document is compared to each of a plurality of signature vectors to determine a primary type of the input document. The primary type of the input document is stored into a storage system in communication with the at least one server.
1. A method, comprising:
receiving, by at least one server communicatively coupled to a network, an input document;
identifying, by the at least one server, a plurality of features in the input document, the plurality of features including sequences of text extracted from the input document;
generating, by the at least one server, a feature vector of the input document based upon the sequences of text;
identifying, by the at least one server, a plurality of signature vectors based upon an input training dataset and at least one cross-type frequency vector;
comparing, by the at least one server, the feature vector of the input document to each of a plurality of signature vectors to determine a primary type of the input document, wherein comparing the feature vector of the input document to each of the plurality of signature vectors to determine the primary type of the input document includes identifying a signature vector that maximizes the expression V·(C t /D), where V is the feature vector, C t is a signature vector t in the plurality of signature vectors, and D is the at least one cross-type frequency vector; and
storing, by the at least one server, the primary type of the input document into a storage system in communication with the at least one server.
2. The method of claim 1 , wherein the plurality of features includes two-gram sequences.
3. The method of claim 1 , including:
identifying a first signature vector in the plurality of signature vectors of the primary type of the input document; and
determining whether (V·(C p /D), where C p is the first signature vector of the primary type of the input document, exceeds a threshold.
4. The method of claim 3 , including, when (V·(C p /D) exceeds the threshold, comparing, by the at least one server, the feature vector to at least one of the plurality of signature vectors to determine a second type of the input document.
5. The method of claim 1 , wherein the input document includes a price list of a business.
6. The method of claim 5 , wherein each one of the plurality signature vectors describes a probability that at least one of the plurality of features appears in a price list having a particular type.
7. The method of claim 1 , wherein the input document is a web page.
8. A system, comprising:
a server computer configured to communicate with a content source using a network, the server computer being configured to:
receive an input document,
identify a plurality of features in the input document, the plurality of features including sequences of text extracted from the input document,
generate a feature vector of the input document based upon the sequences of text,
identify a plurality of signature vectors based upon an input training dataset and at least one cross-type frequency vector,
compare the feature vector of the input document to each of a plurality of signature vectors to determine a primary type of the input document wherein comparing the feature vector of the input document to each of the plurality of signature vectors to determine the primary type of the input document includes identifying a signature vector that maximizes the expression V·(C t /D), where V is the feature vector, C t is a signature vector t in the plurality of signature vectors, and D is the at least one cross-type frequency vector, and
store the primary type of the input document into a storage system.
9. The system of claim 8 , wherein the plurality of features includes two-gram sequences.
10. The system of claim 8 , wherein the server computer is configured to:
identify a first signature vector in the plurality of signature vectors of the primary type of the input document; and
determine whether (V·(C p /D), where C p is the first signature vector of the primary type of the input document, exceeds a threshold.
11. The system of claim 10 , wherein the server computer is configured to, when (V·(C p /D) exceeds the threshold, compare the feature vector to at least one of the plurality of signature vectors to determine a second type of the input document.
12. The system of claim 8 , wherein the input document includes a price list of a business.
13. The system of claim 12 , wherein each one of the plurality signature vectors describes a probability that at least one of the plurality of features appears in a price list having a particular type.
14. The system of claim 8 , wherein the input document is a web page.