Training and testing machine learning models using biased samples
Techniques for training and testing machine learning models using biased samples are described. In some examples, a sampled test set is generated by estimating a propensity score for each annotation of the set of annotations, wherein a propensity score quantifies a likelihood of being human generated using the set of data, estimating a confidence score for each annotation of the set of annotations, wherein a confidence score quantifies a confidence in a correctness of the annotation, mapping each annotation of the set of annotations to a multi-dimensional space based at least in part on the propensity score, stratifying, based on the propensity score, the mapped annotations, and sampling each stratum according to a request to generate a sampled test set.
1 . A computer-implemented method comprising:
receiving audio data to be annotated;
receiving text derived from the audio data;
receiving annotations associated with the text derived from the audio data, wherein at least a proper subset of the annotations is human-generated using the data according to an opt-in preference;
generating a sampled test set by:
estimating, by an audio annotation service implemented as audio annotation service code executed out of memory by one or more processors, a propensity score for each annotation, wherein the propensity score quantifies a likelihood of each annotation being human-generated using the audio data;
estimating a confidence score for each annotation, wherein the confidence score quantifies a confidence in a correctness of the annotation;
mapping each annotation to a multi-dimensional space based at least in part on the propensity score to generate mapped annotations;
binning the mapped annotations; and
sampling each bin of the mapped annotations according to a request to generate the sampled test set; and
training a machine learning (ML) model using the sampled test set.
2 . The computer-implemented method of claim 1 , wherein the request to generate a sampled test set indicates the sampling is to only use the proper subset of the annotations that are human-generated using the data according to an opt-in preference.
3 . The computer-implemented method of claim 1 , wherein the request to generate a sampled test set indicates the sampling is to use the proper subset of the annotations that are human-generated using the data and a proper subset of the annotations that are human-generated without using the data.
4 . A computer-implemented method comprising:
receiving data to be annotated;
receiving annotations associated with the data, wherein at least a proper subset of the annotations is human-generated using the data according to an opt-in preference;
sampling the annotations to generate a sampled test set by:
estimating, by an audio annotation service implemented as audio annotation service code executed out of memory by one or more processors, a propensity score for each annotation, wherein the propensity score quantifies a likelihood of each annotation being human-generated using the data,
estimating a confidence score for each annotation, wherein the confidence score quantifies a confidence in a correctness of the annotation,
mapping each annotation to a multi-dimensional space based at least in part on the propensity score to generate mapped annotations,
stratifying, based on the propensity score, the mapped annotations into strata, and
sampling each stratum of the mapped annotations according to a request to generate the sampled test set; and
performing at least one of evaluating a machine learning (ML) model using the sampled test set or training another ML model using the sampled test set.
5 . The computer-implemented method of claim 4 , wherein the training is performed using a service of a provider network.
6 . The computer-implemented method of claim 5 , wherein the request to generate a sampled test set indicates the sampling is to only use the proper subset of the annotations that are human-generated using the data according to an opt-in preference.
7 . The computer-implemented method of claim 5 , wherein the request to generate a sampled test set indicates the sampling is to use the proper subset of the annotations that are human-generated using the data and a proper subset of the annotations that are human-generated without using the data.
8 . The computer-implemented method of claim 7 , wherein low confidence score annotations from the proper subset of the annotations that are human-generated without using the data are swapped with annotations that are human-generated using the data of a same stratum.
9 . The computer-implemented method of claim 8 , wherein the low confidence score annotations are below a confidence score threshold.
10 . The computer-implemented method of claim 9 , wherein the confidence score threshold is configurable.
11 . The computer-implemented method of claim 5 , wherein evaluating a ML model uses a statistical estimator.
12 . The computer-implemented method of claim 11 , wherein the statistical estimator is to re-weight samples in the sampled test set based at least in part on the propensity scores.
13 . The computer-implemented method of claim 4 , wherein the data to be annotated is audio data.
14 . The computer-implemented method of claim 13 , wherein text is generated from the audio data using at least one automatic speech recognition machine learning model.
15 . A system comprising:
a first one or more electronic devices to implement a data storage service in a multi-tenant provider network; and
a second one or more electronic devices to implement an annotation service in the multi-tenant provider network, the annotation service including instructions that upon execution cause the annotation service to:
receive data to be annotated from the data storage service;
receive annotations associated with the data, wherein at least a proper subset of the annotations is human-generated using the data according to an opt-in preference;
sample the annotations to generate a sampled test set by
estimating, by an audio annotation service implemented as audio annotation service code executed out of memory by one or more processors, a propensity score for each annotation, wherein the propensity score quantifies a likelihood of each annotation being human-generated using the data,
estimating a confidence score for each annotation, wherein the confidence score quantifies a confidence in a correctness of the annotation,
mapping each annotation to a multi-dimensional space based at least in part on the propensity score to generate mapped annotations,
stratifying, based on the propensity score, the mapped annotations into strata, and
sampling each stratum of the mapped annotations according to a request to generate the sampled test set; and
perform at least one of evaluating a machine learning (ML) model using the sampled test set or training another ML model using the sampled test set.
16 . The system of claim 15 , further comprising a model training service to train the ML model using the sampled test set.
17 . The system of claim 15 , further comprising a model hosting service to host a ML model trained using the sampled test set.
18 . The system of claim 15 , wherein the request to generate a sampled test set indicates the sampling is to only use the proper subset of the annotations that are human-generated using the data according to an opt-in preference.
19 . The system of claim 15 , wherein the data to be annotated is audio data.
20 . The system of claim 15 , wherein text is generated from the audio data using at least one automatic speech recognition machine learning model.