Targeted data acquisition for model training
Targeted acquisition of data for model training includes identifying attributes of classified samples of a collection of samples classified by a classification model, and generating at least one query based on the identified attributes, the at least one query tailored, based on the attributes, to retrieve additional training data for training the classification model to more accurately classify samples and avoid incorrect sample classification.
1. A computer-implemented method comprising:
identifying attributes of classified samples of a collection of samples classified by a classification model; and
generating at least one query based on the identified attributes, the at least one query tailored, based on the attributes, to retrieve additional training data for training the classification model to more accurately classify samples and avoid incorrect sample classification.
2. The method of claim 1 , wherein the identified attributes comprises attributes of one or more correctly classified samples of the collection of samples and one or more incorrectly classified samples of the collection of samples.
3. The method of claim 1 , wherein the generating generates a plurality of queries based on the identified attributes.
4. The method of claim 3 , further comprising:
ranking the generated plurality of queries into a ranked list of queries; and
selecting, from the ranked list of queries, a query to issue, wherein the selecting applies a budget comprising one or more constraints on resources to execute the query, the resources comprising at least one selected from the group consisting of: computing cost, memory, time, and electrical power.
5. The method of claim 1 , further comprising:
obtaining an embedding of the collection of samples in an embedding space in which the samples are embedded, the embedding presenting a respective classification of each sample of the collection of samples by the classification model; and
defining neighborhoods of the embedding space, wherein each neighborhood of the neighborhoods comprises a respective at least one incorrectly classified sample of the collection of samples, embedded in the embedding space, that the classification model incorrectly classified, and a respective at least one correctly classified sample of the collection of samples, embedded in the embedding space, that the classification model correctly classified.
6. The method of claim 5 , wherein the defining the neighborhoods comprises using at least one radius to define a region of the embedding space centered around an incorrectly classified sample, wherein the region defines a neighborhood of the neighborhoods, and wherein any samples within the region are samples within that neighborhood.
7. The method of claim 1 , further comprising constructing, using generated metadata, knowledge graphs that inform the identified attributes of one or more correctly classified samples of the collection of samples and one or more incorrectly classified samples of the collection of samples, wherein the constructing the knowledge graphs comprises constructing a respective knowledge graph for each neighborhood of the neighborhoods based on metadata generated for that neighborhood, wherein the method further comprises combining attributes informed by at least some of the constructed knowledge graphs to construct an aggregate knowledge graph, and wherein the generating generates the at least one query from the aggregate knowledge graph.
8. The method of claim 1 , wherein the collection of samples comprises images, wherein at least some of the identified attributes comprise an object depicted in at least some of the images, and wherein the generating the at least one query uses at least one visibility flag, indicating whether the object is visible in the images, in determining whether to tailor the at least one query to include or omit results in which the object is visible.
9. The method of claim 1 , wherein the generated at least one query is implemented as at least one crawl path for crawling at least one remote resource to retrieve the additional training data.
10. The method of claim 1 , wherein a query of the generated at least one query is multimodal, in which the multimodal query queries for both text and image results.
11. The method of claim 1 , further comprising:
issuing the at least one query;
retrieving, in response to the issuing, an additional one or more samples on which to train the classification model; and
retraining the classification model using the additional one or more samples.
12. The method of claim 11 , further comprising, based on the retraining:
iterating one or more times: (i) the identifying attributes, (ii) the generating at least one query, (iii) the issuing the at least one query, (v) the retrieving an additional one or more samples, and (vi) the retraining the classification model; and
checking whether to halt the iterating by testing stability of the classification model and determining whether a threshold has been reached to avoid overfitting the classification model.
13. A computer system comprising:
a memory; and
a processor in communication with the memory, wherein the computer system is configured to perform a method comprising:
identifying attributes of classified samples of a collection of samples classified by a classification model; and
generating at least one query based on the identified attributes, the at least one query tailored, based on the attributes, to retrieve additional training data for training the classification model to more accurately classify samples and avoid incorrect sample classification.
14. The computer system of claim 13 , wherein the identified attributes comprises attributes of one or more correctly classified samples of the collection of samples and one or more incorrectly classified samples of the collection of samples.
15. The computer system of claim 13 , wherein the generating generates a plurality of queries based on the identified attributes.
16. The computer system of claim 13 , wherein the method further comprises:
obtaining an embedding of the collection of samples in an embedding space in which the samples are embedded, the embedding presenting a respective classification of each sample of the collection of samples by the classification model; and
defining neighborhoods of the embedding space, wherein each neighborhood of the neighborhoods comprises a respective at least one incorrectly classified sample of the collection of samples, embedded in the embedding space, that the classification model incorrectly classified, and a respective at least one correctly classified sample of the collection of samples, embedded in the embedding space, that the classification model correctly classified.
17. The computer system of claim 13 , wherein the method further comprises constructing, using generated metadata, knowledge graphs that inform the identified attributes of one or more correctly classified samples of the collection of samples and one or more incorrectly classified samples of the collection of samples, wherein the constructing the knowledge graphs comprises constructing a respective knowledge graph for each neighborhood of the neighborhoods based on metadata generated for that neighborhood, wherein the method further comprises combining attributes informed by at least some of the constructed knowledge graphs to construct an aggregate knowledge graph, and wherein the generating generates the at least one query from the aggregate knowledge graph.
18. The computer system of claim 13 , wherein the collection of samples comprises images, wherein at least some of the identified attributes comprise an object depicted in at least some of the images, and wherein the generating the at least one query uses at least one visibility flag, indicating whether the object is visible in the images, in determining whether to tailor the at least one query to include or omit results in which the object is visible.
19. The computer system of claim 13 , wherein a query of the generated at least one query is multimodal, in which the multimodal query queries for both text and image results.
20. The computer system of claim 13 , wherein the method further comprises:
issuing the at least one query;
retrieving, in response to the issuing, an additional one or more samples on which to train the classification model; and
retraining the classification model using the additional one or more samples.
21. A computer program product comprising:
a computer readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising:
identifying attributes of classified samples of a collection of samples classified by a classification model; and
generating at least one query based on the identified attributes, the at least one query tailored, based on the attributes, to retrieve additional training data for training the classification model to more accurately classify samples and avoid incorrect sample classification.
22. The computer program product of claim 21 , wherein the identified attributes comprises attributes of one or more correctly classified samples of the collection of samples and one or more incorrectly classified samples of the collection of samples.
23. The computer program product of claim 21 , wherein the generating generates a plurality of queries based on the identified attributes.
24. The computer program product of claim 21 , wherein the method further comprises:
obtaining an embedding of the collection of samples in an embedding space in which the samples are embedded, the embedding presenting a respective classification of each sample of the collection of samples by the classification model; and
defining neighborhoods of the embedding space, wherein each neighborhood of the neighborhoods comprises a respective at least one incorrectly classified sample of the collection of samples, embedded in the embedding space, that the classification model incorrectly classified, and a respective at least one correctly classified sample of the collection of samples, embedded in the embedding space, that the classification model correctly classified.
25. The computer program product of claim 21 , wherein the method further comprises:
issuing the at least one query;
retrieving, in response to the issuing, an additional one or more samples on which to train the classification model; and
retraining the classification model using the additional one or more samples.