Method and system for developing a classification tool
View Patent ↗An exemplary embodiment of the present invention provides a computer implemented method of developing a classifier. The method includes receiving input for a case, the case comprising a plurality of instances and an example, the example comprising a plurality of data fields each corresponding to one of the plurality of instances, wherein the input indicates which, if any, of the instances includes a data field belonging to a target class. The method also includes training the classifier based, at least in part, on the input from the trainer.
1. A computer-implemented method of developing a classifier, comprising:
receiving input for a case, the case comprising a plurality of instances and an example, the example comprising a plurality of data fields each corresponding to one of the plurality of instances, wherein the input indicates which one, if any, of the plurality of instances includes a data field belonging to a target class; and
training the classifier based, at least in part, on the input from the trainer.
2. The computer-implemented method of claim 1 , comprising obtaining a collection of electronic data comprising the data fields, and grouping the electronic data into cases comprising one or more examples with corresponding instances.
3. The computer-implemented method of claim 1 , wherein the case comprises a plurality of examples corresponding with query URLs generated by a plurality of users.
4. The computer-implemented method of claim 1 , wherein the target class comprises those data fields that include search terms entered by a user.
5. The computer-implemented method of claim 2 , wherein the electronic data comprises a plurality of query URLs and grouping the electronic data into cases comprises grouping together query URLs that are determined to be similar.
6. The computer-implemented method of claim 5 , wherein the query URLs are determined to be similar based on a number of data field names in common.
7. The computer-implemented method of claim 1 , comprising using the classifier to generate a score for an unlabeled instance based, at least in part, on an instance feature, the score corresponding to a likelihood that the unlabeled instance is of a target class.
8. The computer-implemented method of claim 7 , wherein presenting the case to the trainer comprises generating a visual indication of the score computed for each instance.
9. The computer-implemented method of claim 1 , wherein presenting the case to the trainer comprises selecting the case based, at least in part, on the number of examples included in the case.
10. The computer-implemented method of claim 1 , wherein the electronic data comprises a plurality of electronic documents, a data field comprises an identifiable portion of a document, and the target class comprises a representation of at least one of an author, a title, a manufacturer, a model, a company, a stock identifier, a mutual fund identifier, a price, an item identifier, a category identifier, an article reference, an advertisement, a non-boilerplate text section, or a portrait photograph.
11. A computer system, comprising:
a processor that is configured to execute machine-readable instructions;
a storage device that is configured to store a case comprising a plurality of instances and an example, the example comprising a plurality of data fields each corresponding to one of the plurality of instances;
a memory device that stores instructions that are executable by the processor, the instructions comprising:
a training system configured to receive an instance label for each of the case's instances according to whether the corresponding instance belongs to a target class, and generate a classifier based, at least in part, on the instance label.
12. The computer system of claim 11 , comprising a case generator configured to receive a collection of electronic data comprising the data fields and group the data fields into a plurality of cases, each case comprising a plurality of examples with corresponding instances.
13. The computer system of claim 11 , comprising a plurality of case viewers running on a plurality of client systems, wherein the training system is configured to receive instance labels from the plurality of case viewers.
14. The computer system of claim 11 , wherein the training system is configured to rank the case based on the number of examples included in the case and provide the case to the case viewer based on the rank.
15. The computer system of claim 11 , wherein the training system is configured to generate a score for each of the instances based, at least in part, on an instance feature, the score corresponding with a degree of likelihood that that the instance is of the target class.
16. The computer system of claim 15 , wherein the instance feature comprises at least one of a minimum, maximum, median, mean, or standard deviation of an individual string feature over the data field values within the instance.
17. The computer system of claim 15 , wherein the training system is configured to compute an uncertainty value based, at least in part, on the score, and provide the case to the case viewer based, at least in part, on the uncertainty value.
18. A non-transitory computer-readable medium, comprising code configured to direct a processor to:
receiving labels for a case, the case comprising a plurality of instances and an example, the example comprising a plurality of data fields each corresponding to one of the plurality of instances, wherein the labels indicate which, if any, of the instances includes a data field belonging to a target class; and
generate a classifier based, at least in part, on the labels.
19. The non-transitory computer-readable medium of claim 18 , comprising code configured to direct a processor to generate an uncertainty value for each of the instances based, at least in part, on an instance feature and provide a second case to the trainer based, at least in part, on the uncertainty value.
20. The non-transitory computer-readable medium of claim 18 , comprising code configured to direct a processor to generate a score for each instance based, at least in part, on an instance feature and flag an unlabeled instance as belonging to a target class based, at least in part, on the score.