Classifier validation

1. A method, comprising:

executing, on a processor of a computing device, instructions that cause the computing device to perform operations, the operations comprising:

partitioning a set of in-sample examples into a reduced in-sample set of examples and a remaining in-sample set of examples;

processing the reduced in-sample set using a set of classifiers;

identifying a subset of classifiers of the set of classifiers as having error counts below a threshold number of errors over the reduced in-sample set;

executing a training procedure to select a classifier from the set of classifiers based upon the classifier having an error rate, over the set of in-sample examples, less than an error rate threshold; and

upon the classifier being in the subset of classifiers, determining an out-of-sample error bound for the classifier based upon an error rate of the classifier over the remaining in-sample set.

2. The method of claim 1 , wherein the determining the out-of-sample error bound comprises:

determining the out-of-sample error bound based upon a count of remaining examples within the remaining in-sample set.

3. The method of claim 1 , wherein the determining the out-of-sample error bound comprises:

determining the out-of-sample error bound based upon a ratio of a selected upper bound for a probability of bound failure and a number of classifiers in the subset of classifiers.

4. The method of claim 1 , comprising:

reporting a bound failure based upon the classifier not being in the subset of classifiers.

5. The method of claim 1 , wherein the identifying a subset of classifiers comprises:

in response to the set of classifiers exceeding a threshold number of classifiers, sampling a portion of the set of classifiers as a set of sampled classifiers for identifying the subset of classifiers.

6. The method of claim 5 , wherein the identifying a subset of classifiers comprises:

identifying a fraction of the set of sampled classifiers that have error counts below the threshold number of errors over the reduced in-sample set.

7. The method of claim 6 , wherein the identifying a subset of classifiers comprises:

utilizing the fraction of the set of sampled classifiers to bound a rate at which classifiers in the set of classifiers have error counts below the threshold number of errors over the reduced in-sample set.

8. The method of claim 1 , wherein the identifying a subset of classifiers comprises:

in response to the set of classifiers not exceeding a threshold number of classifiers, testing all classifiers within the set of classifiers for identifying classifiers having error counts below the threshold number of errors over the reduced in-sample set.

9. The method of claim 1 , comprising:

in response to the out-of-sample error bound being within a tolerance threshold, utilizing the classifier to classify out-of-sample data.

10. The method of claim 1 , comprising:

in response to the out-of-sample error bound exceeding a tolerance threshold, obtaining additional in-sample examples for training the set of classifiers.

11. A non-transitory machine readable medium having stored thereon processor-executable instructions that when executed cause performance of operations, the operations comprising:

partitioning a set of in-sample examples into a reduced in-sample set of examples and a remaining in-sample set of examples;

processing the reduced in-sample set using a set of classifiers;

identifying a subset of classifiers of the set of classifiers as having error counts below a threshold number of errors over the reduced in-sample set;

executing a training procedure to select a classifier from the set of classifiers based upon the classifier having a minimum error rate over the set of in-sample examples; and

upon the classifier being in the subset of classifiers, determining an out-of-sample error bound for the classifier based upon an error rate of the classifier over the remaining in-sample set, a count of remaining examples within the remaining in-sample set, and a ratio of a selected upper bound for a probability of bound failure and a number of classifiers in the subset of classifiers.

12. The non-transitory machine readable medium of claim 11 , wherein the operations comprise:

in response to the out-of-sample error bound being within a tolerance threshold, utilizing the classifier to classify out-of-sample data.

13. A computing device comprising:

a processor; and

memory comprising processor-executable instructions that when executed by the processor cause performance of operations, the operations comprising:

partitioning a set of in-sample examples into a reduced in-sample set of examples and a remaining in-sample set of examples;

determining an upper bound on a probability that a classifier drawn according to a distribution has an error count below a threshold number of errors over the reduced in-sample set;

performing a training procedure for weighted ensemble classifiers to select a posterior distribution over a set of classifiers using the set of in-sample examples; and

computing an out-of-sample error bound based upon a divergence between the distribution and the posterior distribution.

14. The computing device of claim 13 , wherein the probability is a shrinkage ratio, and wherein the computing an out-of-sample error bound comprises:

adding a natural logarithm of the upper bound as the shrinkage ratio to the divergence.

15. The computing device of claim 13 , wherein the computing an out-of-sample error bound comprises:

utilizing the remaining in-sample set as in-sample examples for computing the out-of-sample error bound.

16. The computing device of claim 13 , wherein the computing an out-of-sample error bound comprises:

adding a fraction of the posterior distribution placed on classifiers that do not have error counts below the threshold number of errors over the reduced in-sample set to the out-of-sample error bound.

17. The computing device of claim 13 , wherein the out-of-sample error bound is a PAC-Bayes out-of-sample error bound.

18. The computing device of claim 13 , wherein the out-of-sample error bound is based upon a Kullback-Leibler divergence.

19. The computing device of claim 13 , wherein the operations comprise:

upon a bound being within an acceptance threshold, utilizing the classifier on out-of-sample data.

20. The computing device of claim 13 , wherein the operations comprise:

upon a bound not being within an acceptance threshold, obtaining additional in-sample data.