Method and system for developing a classification tool
View Patent ↗An exemplary embodiment of the present invention provides a computer implemented method of developing a classifier. The method includes obtaining a set of training data comprising labeled cases. The method also includes training a classifier based, at least in part, on the training data. The method also includes applying the classifier to a plurality of unlabeled cases to generate classification scores for each of the unlabeled cases, wherein each classification score corresponds with an instance of a corresponding case. Furthermore, the classification score corresponding to a first instance in a case is computed based, at least in part, on a value of a case-centric feature corresponding to the first instance, wherein the value of the case-centric feature is based, at least in part, on characteristics of the first instance and a second instance in the case.
1. A computer implemented method of generating a classifier, comprising:
obtaining a set of training data comprising labeled cases;
training a classifier based, at least in part, on the training data;
applying the classifier to a plurality of unlabeled cases to generate classification scores for each of the unlabeled cases, wherein each classification score corresponds with an instance of a corresponding case; and
wherein the classification score corresponding to a first instance in a case is computed based, at least in part, on a value of a case-centric feature corresponding to the first instance, wherein the value of the case-centric feature is based, at least in part, on characteristics of the first instance and a second instance in the case.
2. The computer implemented method of claim 1 , comprising:
generating a desirability factor for one or more of the unlabeled cases, based, at least in part, on the classification scores, the desirability factor corresponding to a level of desirability of selecting a corresponding case as a next case for which to obtain training data; and
selecting one of the unlabeled cases as the next case for which to obtain input based, at least in part, on the desirability factor.
3. The computer implemented method of claim 1 , wherein computing the value of the case-centric feature comprises normalizing a value of a second feature corresponding to the first instance with respect to values of the second feature corresponding to a plurality of instances in the case.
4. The computer implemented method of claim 1 , wherein computing the value of the case-centric feature comprises ranking a plurality of instances in the case according to the values of a second feature corresponding to each of the plurality of instances.
5. The computer implemented method of claim 1 , wherein the case-centric feature is based, at least in part, on a name associated with the first instance and a name associated with a second instance in the case.
6. The computer implemented method of claim 1 , wherein generating the desirability factor comprises generating an uncertainty value for each instance in the case, the uncertainty value is based, at least in part, on the proximity of the classification score to a classification threshold, and the desirability factor is based, at least in part, on the uncertainty values.
7. The computer implemented method of claim 6 , wherein generating the desirability factor comprises summing the uncertainty values.
8. The computer implemented method of claim 6 , wherein generating the desirability factor comprises ranking the instances in the case.
9. The computer implemented method of claim 1 , wherein the desirability factor is modified by a weighting factor that is based, at least in part, on a similarity between cases.
10. The computer implemented method of claim 9 , wherein the weighting factor is based, at least in part, on a distance between the instances of a first case and the instances of a second case.
11. The computer implemented method of claim 1 , comprising generating one or more case-centric evaluation parameters that characterize the quality of the classifier.
12. The computer implemented method of claim 11 , wherein generating one or more case-centric evaluation parameters comprises recording a correct positive tally, a correct negative tally, a missed positive tally, a missed negative tally, and a wrong positive tally, and the case centric evaluation parameters are based, at least in part, on one or more of the tallies.
13. A computer system, comprising:
a processor that is configured to execute machine-readable instructions;
a memory device that is configured to store a classifier, a set of training data comprising labeled cases, and instructions that are executable by the processor, the instructions comprising:
a score generator configured to apply the classifier to generate classification scores for each of the unlabeled cases; each classification score corresponding with an instance of the unlabeled case;
a desirability generator configured to generate a desirability factor for one or more of the unlabeled cases, based, at least in part, on the classification scores, the desirability factor corresponding to a level of desirability of selecting the corresponding case as the next case for which to obtain training data; and
a case selector configured to select one of the unlabeled cases as the next case for which to obtain input based, at least in part, on the desirability factor.
14. The computer system of claim 13 , comprising an evaluator configured to characterize a quality of the classifier by recording a correct-positive tally, a correct-negative tally, a missed-positive tally, a missed-negative tally, and a wrong-positive tally, and generating case centric evaluation parameters based, at least in part, on one or more of the tallies.
15. The computer system of claim 13 , wherein the classification score corresponding to a first instance in a case is computed based, at least in part, on a value of a case-centric feature corresponding to the first instance, wherein the value of the case-centric feature is based, at least in part, on characteristics of the first instance and a second instance in the case.
16. The computer system of claim 13 , wherein the desirability generator is configured to compute the desirability factor by generating an uncertainty value for each instance in the case and summing the uncertainty values.
17. The computer system of claim 16 , wherein the desirability generator is configured to modify the desirability factor of a first case by a weighting factor that is computed based, at least in part, on a Euclidean distance between the instances of the first case and the instances of a second case.
18. A tangible, non-transitory, computer-readable medium, comprising code configured to direct a processor to:
generate a classification score for an instance of an unlabeled case;
generate a desirability factor for the unlabeled case, based, at least in part, on the classification score, the desirability factor corresponding to a level of desirability of selecting the unlabeled case as the next case for which to obtain training data; and
select the unlabeled case as the next case for which to obtain input based, at least in part, on the desirability factor.
19. The tangible, non-transitory, computer-readable medium of claim 18 , comprising code configured to direct the processor to generate a case-centric instance feature, wherein the classifications scores are generated based, at least in part, on the case-centric instance feature.
20. The tangible, non-transitory, computer-readable medium of claim 18 , comprising code configured to direct the processor to generate an uncertainty value for the instance based, at least in part, on the proximity of the classification score to the classification threshold, and the desirability factor is based, at least in part, on the uncertainty values.