Predictive model evaluation and training based on utility

1. A computer-implemented method, the method comprising:

training a plurality of predictive models using training data and respective different machine learning techniques, the training data including a plurality of examples that are each associated with an answer that corresponds to one of multiple answer categories;

obtaining different weights for each combination of two of the answer categories that correspond to the answers in the plurality of mimics in the framing data;

calculating a weighted accuracy score for each predictive model, comprising:

performing a plurality of rounds of cross-validation of the predictive model using the training data, wherein each round of cross-validation is performed using at least one of K-fold validation, k×2 cross-validation, or random sub-sampling validation, to produce a plurality of answers predicated for corresponding examples in the training data; and

for each of the predictive models, for one or more of the plurality of answers predicted for the examples, applying the respective weight that corresponds to the combination of two of the answer categories that matches the answer category that corresponds to the answer of example as indicated by the training data and the answer category of the answer predicted by the predictive model for the example; and

selecting one of the predictive models as the most accurate model out of the plurality of predictive models based at least partly on the calculated weighted accuracy scores.

2. The method of claim 1 , wherein a particular answer of the answers includes at least one of a label, a numeric value, a range of numeric values, or a set of numeric values.

3. The method of claim 1 wherein the one or more of the plurality of answers predicted are correct.

4. The method of claim 1 wherein the one or more of the plurality of answers predicted are incorrect.

5. The method of claim 1 wherein the weight associated with each combination of two of the answer categories that correspond to the answers in the plurality of examples in the training data is based at least partly on the training data.

6. The method of claim 1 wherein obtaining different weights for each combination of two of the answer categories that correspond to the answers in the plurality of examples in the training data comprises:

obtaining the respective different weights based on a distribution of answer categories that correspond to the answers of the examples in the training data.

7. The method of claim 1 wherein the plurality of predictive models include a Naive Bayes classifier, a k-nearest neighbor classifier, a support vector machine, or a predictive model that uses a logistic regression technique.

8. The method of claim 1 , wherein obtaining different weights for each combination of two of the answer categories that correspond to the answers in the plurality of examples in the training data comprises:

receiving a single user specified weight for a particular answer category; and

obtaining the different weights for each combination of the particular answer category and another answer category of the answer categories that correspond to the answers in the plurality of examples in the training data based on the single user specified weight.

9. The method of claim 1 , wherein obtaining different weights for each combination of two of the answer categories that correspond to the answers in the plurality of examples in the training data comprises:

receiving a first user specified weights for a particular answer category and a second user specified weights for the particular answer category;

obtaining the different weights for each combination of the particular answer category as a false positive based on the first user specified weight; and

obtaining the different weights for each combination of the particular answer category as a false negative based on the second user specified weight.

10. The method of claim 1 , comprising:

storing the weights with the plurality of predictive models; and

reusing the weights stored with the plurality of predictive models to retrain the plurality of predictive models with new training data.

11. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

training a plurality of predictive models training data and respective different machine learning techniques, the training data including a plurality of examples that are each associated with an answer that corresponds to one of the multiple answer categories;

obtaining different weights for each combination of two of the answer categories that correspond to the answers in the plurality of examples in the training data;

calculating a weighted accuracy score for each predictive model, comprising:

performing a plurality of rounds of cross-validation of the predictive model using the training data, wherein each round of cross-validation is performed using at least one of K-fold validation, k×2 cross-validation, or random sub-sampling validation, to produce a plurality of answers predicated for corresponding examples in the training data; and

for each of the predictive models, for one or more of the plurality of answers predicted for the examples, applying the respective weight that corresponds to the combination of two of the answer categories that matches the answer category that corresponds to the answer of example as indicated by the training data and the answer category of the answer predicted by the predictive model for the example; and

selecting one of the predictive models as the most accurate model out of the plurality of predictive models based at least partly on the calculated weighted accuracy scores.

12. The system of claim 11 , wherein a particular answer of the answers includes at least one of a label, a numeric value, a range of numeric values, or a set of numeric values.

13. The system of claim 11 wherein the one or more of the plurality of answers predicted are correct.

14. The system of claim 11 wherein the one or more of the plurality of answers predicted are incorrect.

15. The system of claim 11 wherein the weight associated with each combination of two of the answer categories that correspond to the answers in the plurality of examples in the training data is based at least partly on the training data.

16. The system of claim 11 wherein obtaining different weights for each combination of two of the answer categories that correspond to the answers in the plurality of examples in the training data comprises:

obtaining the respective different weights based on a distribution of answer categories that correspond to the answers of the examples in the training data.

17. The system of claim 11 wherein the plurality of predictive models include a Naive Bayes classifier, a k-nearest neighbor classifier, a support vector machine, or a predictive model that uses a logistic regression technique.

18. A non-transitory computer-readable medium storing instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

training a plurality of predictive models using training data and respective different machine learning techniques, the training data including a plurality of examples that are each associated with an answer that corresponds to one of multiple answer categories;

obtaining different weights for each combination of two of the answer categories that correspond to the answers in the plurality of mimics in the framing data;

calculating a weighted accuracy score for each predictive model, comprising:

performing a plurality of rounds of cross-validation of the predictive model using the training data, wherein each round of cross-validation is performed using at least one of K-fold validation, k×2 cross-validation, or random sub-sampling validation, to produce a plurality of answers predicated for corresponding examples in the training data; and

for each of the predictive models, for one or more of the plurality of answers predicted for the examples, applying the respective weight that corresponds to the combination of two of the answer categories that matches the answer category that corresponds to the answer of example as indicated by the training data and the answer category of the answer predicted by the predictive model for the example; and

selecting one of the predictive models as the most accurate model out of the plurality of predictive models based at least partly on the calculated weighted accuracy scores.

19. The medium of claim 18 , wherein a particular answer of the answers includes at least one of a label, a numeric value, a range of numeric values, or a set of numeric values.