Classifier

1. A method comprising, by at least one processor:

receiving a training dataset of cases, wherein the cases in the training dataset are associated with a plurality of classes;

selecting a first group of the plurality of classes in the training dataset to be a proper subset of classes for training a first classifier model;

selecting a second group of the plurality of classes that are excluded from the proper subset of classes to be an excluded group of classes for testing the first classifier model;

selecting a partition of the cases in the training dataset to be a test partition for testing the first classifier model;

training the first classifier model on the proper subset of classes using the cases in the training dataset that are not in the test partition and not associated with the excluded group of classes;

testing the first classifier model on the excluded group of classes using the cases in the test partition; and

determining a performance measurement of the first classifier model.

2. The method of claim 1 , further comprising, by the at least one processor:

determining a performance measurement of a second classifier model;

comparing the performance measurement of the first classifier model and the second. classifier model; and

selecting a best performing classifier model based on the comparison.

3. The method of claim 1 , wherein the proper subset of classes is randomly selected from the plurality of classes in the training dataset.

4. The method of claim 1 , wherein the plurality of classes in the training dataset include at least three classes.

5. The method of claim 1 , further comprising, by the at least one processor:

partitioning the plurality of classes in the training dataset into a plurality of groups, including the proper subset of classes and the excluded group of classes.

6. The method of claim 5 , further comprising, by the at least one processor:

partitioning the cases in the training dataset into a plurality of partitions, including the test. partition.

7. The method of claim 1 , wherein the excluded group of classes represents classes that are unfamiliar to the first classifier model.

8. The method of claim 1 , further comprising, by the at least one processor:

determining the performance measurement of the first classifier model for a plurality of test folds;

aggregating the performance measurement of the first classifier model from the plurality of test folds; and

outputting the aggregated performance measurement of the first classifier model.

9. A system comprising:

a processor; and

a memory storing instructions that when executed by the processor cause the processor to:

receive a training dataset of cases, wherein the cases in the training dataset are associated with a plurality of classes,

select a first group of the plurality of classes in the training dataset to be a proper subset of classes for training a first classifier model,

select a second group of the plurality of classes that are excluded from the proper subset of classes to be an excluded group of classes for testing the first classifier model,

select a partition of the cases in the training dataset to be a test partition for testing the first classifier model,

train the first classifier model on the proper subset of classes using the cases in the training dataset that are not in the test partition and not associated with the excluded group of classes,

test the first classifier model on the excluded group of classes using the cases in the test partition, and

determine a performance measurement of the first classifier model.

10. The system of claim 9 , wherein the instructions further cause the processor to:

partition the cases in the training dataset into a plurality of partitions, including the test partition; and

partition the plurality of classes in the training dataset into a plurality of groups, including the proper subset of classes and the excluded group of classes.

11. The system of claim 9 , wherein the excluded group of classes represents classes that are unfamiliar to the first classifier model.

12. The system of claim 9 , wherein the instructions further cause the processor to:

determine the performance measurement of the first classifier model for a plurality of test folds;

aggregate the performance measurement of the first classifier model from the plurality of test folds; and

output the aggregated performance measurement of the first classifier model.

13. The system of claim 9 , wherein the instructions further cause the processor to:

determine a performance measurement of a second classifier model;

compare the performance measurement of the first classifier model and the second classifier model; and

select a best performing classifier model based on the comparison.

14. A non-transitory machine-readable storage medium storing instructions that when executed by at least one processor cause the at least one processor to:

receive a training dataset of cases, wherein the cases in the training dataset are associated with a plurality of classes;

select a first group of the plurality of classes in the training dataset to be a proper subset of classes for training a first classifier model;

select a second group of the plurality of classes that are excluded from the proper subset of classes to be an excluded group of classes for testing the first classifier model;

select a partition of the cases in the training dataset to be a test partition for testing the first classifier model;

train the first classifier model on the proper subset of classes using the cases in the training dataset that are not in the test partition and not associated with the excluded group of classes;

test the first classifier model on the excluded group of classes using the cases in the test partition; and

determine a performance measurement of the first classifier model.

15. The non-transitory machine-readable storage medium of claim 14 , wherein the instructions further cause the at least one processor to:

determine a performance measurement of a second classifier model;

compare the performance measurement of the first classifier model and the second classifier model; and

select a best performing classifier model based on the comparison.

16. The non-transitory machine-readable storage medium of claim 14 , wherein the excluded group of classes represents classes that are unfamiliar to the first classifier model.

17. The non-transitory machine-readable storage medium of claim 14 , wherein the proper subset of classes is randomly selected from the plurality of classes.

18. The non-transitory machine-readable storage medium of claim 14 , wherein the test partition is randomly selected from the cases in the training dataset.

19. The non-transitory machine-readable storage medium of claim 14 , wherein the instructions further cause the at least one processor to:

partition the plurality of classes in the training dataset into a plurality of groups, including the proper subset of classes and the excluded group of classes.

20. The non--transitory machine-readable storage medium of claim 14 , wherein the instructions further cause the at least one processor to:

partitioning the cases in the training dataset into a plurality of partitions, including the test partition.