Method, apparatus, and computer program product for classification and tagging of textual data
Provided herein are systems, methods and computer readable media for classification and tagging of textual data. An example method may include accessing a corpus comprising a plurality of documents, each document having one or more labels indicative of services offered by a merchant, generating a query based on extracted features and the documents, generating a precision score for at least a portion of the generated query and selecting a subset of the generated queries based on an assigned precision score satisfying a precision score threshold, the selected subset of the generated queries configured to provide an indication of one or more labels to be applied to machine readable text. A second example method, utilized for tagging machine readable text with unknown labels, may include assigning a label to textual portions of the machine readable text based on results of the application of the queries.
1. A method for labeling machine-readable text as pertaining to one or more services in a service taxonomy, the method comprising:
accessing the machine-readable text, wherein at least a portion of the machine-readable text is accessed from a merchant website;
applying one or more queries to the machine-readable text, wherein the one or more queries are generated by:
accessing a corpus comprising a plurality of documents, each of one or more documents of the corpus having one or more labels indicative of one or more services offered by a merchant;
generating one or more queries based on one or more extracted features and the one or more documents;
generating a precision score for at least a portion of the generated one or more queries; and
selecting a subset of queries from the generated one or more queries that satisfy a precision score threshold; and
assigning, using a processor, a label to textual portions of the machine-readable text based on results of an application of the subset of queries to the machine-readable text; and
classifying a merchant based on the label.
2. The method according to claim 1 , wherein generating the one or more queries further comprises:
generating an array of feature index pairs, the array of feature index pairs comprising one or more features and a position of the one or more features in a sentence;
generating the one or more queries as a function of one or more combinations of feature index pairs based on the array of feature index pairs; and
outputting the one or more queries.
3. The method according to claim 2 , wherein generating the one more or queries further comprises:
calculating a distance between a first feature in a query and a second feature in the query; and
generating a distance measure for the query.
4. The method according to claim 3 , the method further comprising:
rounding the distance between the first feature and the second feature to a next highest multiple of a predetermined number.
5. The method according to claim 1 , wherein assigning a label to textual portions of the machine-readable text based on results of the application of the subset of queries to the machine-readable text further comprises:
generating a score for the machine-readable text, wherein the score is a function of the precision score of a query divided by a normalization factor, the normalization factor being a function of a subset of one or more precision scores; and generating at least one label for the machine-readable text.
6. An apparatus for labeling machine-readable text as pertaining to one or more services, the machine-readable text recovered from one or more electronic sources, the apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least:
access the machine-readable text, wherein at least a portion of the machine-readable text is accessed from a merchant website;
apply one or more queries to the machine-readable text, wherein the one or more queries are generated by previously causing the apparatus to at least:
access a corpus comprising a plurality of documents, each of one or more documents of the corpus having one or more labels indicative of one or more services offered by a merchant;
generate one or more queries based on one or more extracted features and the one or more documents;
generate a precision score for at least a portion of the generated one or more queries; and
select a subset of queries from the generated one or more queries that satisfy a precision score threshold; and
assign a label to textual portions of the machine-readable text based on results of an application of the subset of queries to the machine-readable text; and
classify a merchant based on the label.
7. The apparatus according to claim 6 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus to:
generate an array of feature index pairs, the array of feature index pairs comprising one or more features and a position of the one or more features in a sentence;
generate the one or more queries as a function of one or more combinations of feature index pairs based on the array of feature index pairs; and
output the one or more queries.
8. The apparatus according to claim 7 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus to:
calculate a distance between a first feature in a query and a second feature in the query; and
generate a distance measure for the query.
9. The apparatus according to claim 8 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus to:
the distance between the first feature and the second feature to a next highest multiple of a predetermined number.
10. The apparatus according to claim 6 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus to:
generate a score for the machine-readable text, wherein the score is a function of the precision score of a query divided by a normalization factor, the normalization score being a function of a subset of one or more precision scores; and generating at least one label for the machine-readable text.
11. A computer program product for labeling machine-readable text as pertaining to one or more services, the machine-readable text recovered from one or more electronic sources, the computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code portions stored therein, the computer-executable program code portions comprising program code instructions for:
accessing the machine-readable text, wherein at least a portion of the machine-readable text is accessed from a merchant website;
applying one or more queries to the machine-readable text, wherein the one or more queries are generated by:
accessing a corpus comprising a plurality of documents, each of one or more documents of the corpus having one or more labels indicative of one or more services offered by a merchant;
generating one or more queries based on one or more extracted features and the one or more documents;
generating a precision score for at least a portion of the generated one or more queries; and
selecting a subset of queries from the generated one or more queries that satisfy a precision score threshold; and
assigning, using a processor, a label to textual portions of the machine-readable text based on results of an application of the subset of queries to the machine-readable text; and
classifying a merchant based on the label.
12. The computer program product according to claim 11 , wherein generating the one or more queries further comprises:
generating an array of feature index pairs, the array of feature index pairs comprising one or more features and a position of the one or more features in a sentence;
generating the one or more queries as a function of one or more combinations of feature index pairs based on the array of feature index pairs; and
outputting the one or more queries.
13. The computer program product according to claim 12 , wherein generating the one or more queries further comprises:
calculating a distance between a first feature in a query and a second feature in the query; and
generating a distance measure for the query.
14. The computer program product according to claim 13 , wherein the computer-executable program code portions further comprise program code instructions for:
rounding the distance between the first feature and the second feature to a next highest multiple of a predetermined number.
15. The computer program product according to claim 11 , wherein assigning a label to textual portions of the machine-readable text based on results of the application of the subset of queries to the machine-readable text further comprises:
generating a score for the machine-readable text, wherein the score is a function of the precision score of a query divided by a normalization factor, the normalization factor being a function of a subset of one or more precision scores; and generating at least one label for the machine-readable text.