Determining an obverse weight

1. A method, comprising:

defining a plurality of bins;

dividing a set of labeled cases into the plurality of bins; and

determining an obverse weight for a bin of the plurality of bins based on (1) an importance weight of the bin and (2) a variance of an error estimate of a classifier on the set of labeled cases in the bin.

2. The method of claim 1 , further comprising:

determining an obverse weight for each bin of the plurality of bins;

selecting at least one case from a plurality of population cases based on the obverse weights of the plurality of bins; and

adding the selected at least one case to the set of labeled cases to yield an augmented set of labeled cases.

3. The method of claim 2 , further comprising:

determining an estimate of a generalization error of the classifier by calculating a weighted average of the number of errors generated by the classifier on the augmented set of labeled cases, weighted by the importance weight of each bin.

4. The method of claim 2 , wherein selecting the at least one case is performed by selecting the case from a bin of the plurality of bins having the highest obverse weight.

5. The method of claim 2 , wherein selecting the at least one case is performed by selecting the case from the plurality of bins according to a probability distribution based on the obverse weights of the plurality of bins.

6. The method of claim 2 , further comprising:

requesting that the selected at least one case be labeled by an expert before adding it to the set of labeled cases.

7. The method of claim 1 , wherein determining the obverse weight for the bin is performed by multiplying the square of the importance weight of the bin by the variance of the bin.

8. The method of claim 1 , further comprising determining the importance weight of the bin based on a proportion of population cases in the bin.

9. The method of claim 8 , wherein determining the importance weight of the bin is performed by dividing a determined number of population cases in the bin by a determined number of labeled cases in the bin.

10. The method of claim 1 , further comprising determining the error estimate of the classifier on the labeled cases in the bin by:

determining a number of errors generated by the classifier on the labeled cases in the bin; and

dividing the number of errors by the number of labeled cases in the bin.

11. The method of claim 1 , further comprising:

selecting a feature of the set of labeled cases having a greatest variation in classification error across the bins,

wherein dividing the set of labeled cases into bins is performed based on a value of the selected feature for each case.

12. The method of claim 1 , wherein dividing the set of labeled cases into bins is performed by applying at least one of a multi-dimensional discretization algorithm and a clustering algorithm to feature vectors of the cases.

13. The method of claim 1 , wherein dividing the set of labeled cases into bins is performed based on an output of the classifier for each case.

14. The method of claim 1 , further comprising:

determining an obverse weight for each bin of the plurality of bins;

defining a second plurality of bins;

dividing the set of labeled cases into a second plurality of bins;

determining a second obverse weight of each bin of the second plurality of bins;

for each case in a set of population cases, aggregating the case's obverse weight and second obverse weight; and

adding the population case having the highest aggregated obverse weight to the set of labeled cases.

15. A system, comprising:

a binning module to divide a set of cases into a plurality of bins;

an obverse weight module to determine an obverse weight of each bin of the plurality of bins based on an importance weight of the bin and a variance of an error estimate of the bin; and

a selection module to select for labeling a case from one of the plurality of bins based on the obverse weights of the plurality of bins.

16. The system of claim 12 , further comprising:

a labeling module to assign a label to the selected case and add the labeled case to a set of test cases.

17. The system of claim 12 , wherein the obverse weight module is configured to determine the obverse weight of each bin by multiplying the square of the importance weight of the bin by the variance of the error estimate of the bin.

18. A non-transitory computer readable storage medium storing instructions that, when executed by a processor, cause a computer to:

divide a set of cases into a plurality of bins; and

determine an obverse weight for a bin of the plurality of bins based on an importance weight of the bin and a variance of an error estimate of the bin.