System and method for efficiently managing large datasets for training an AI model
Embodiments described herein provide a system for facilitating efficient dataset management. During operation, the system obtains a first dataset comprising a plurality of elements. The system then determines a set of categories for a respective element of the plurality of elements by applying a plurality of AI models to the first dataset. A respective category can correspond to an AI model. Subsequently, the system selects a set of sample elements associated with a respective category of a respective AI model and determines a second dataset based on the selected sample elements.
1. A method for facilitating efficient dataset management, comprising:
obtaining a first dataset comprising a plurality of data elements for training a first artificial intelligence (AI) model;
determining respective sets of categories of data elements by applying a plurality of AI models to the first dataset, wherein a respective set of categories corresponds to an AI model;
determining a set of joint categories from the sets of categories, wherein a respective joint category corresponds to at least two categories from at least two sets of categories, respectively;
selecting a set of sample data elements associated with a respective category of a respective set of categories by obtaining the set of sample data elements from the set of joint categories;
determining a second dataset based on the selected sample data elements; and
training the first AI model using the second dataset.
2. The method of claim 1 , wherein the plurality of AI models includes one or more pre-trained classifiers.
3. The method of claim 1 , wherein applying an AI model of the plurality of AI models to the first dataset comprises categorizing the plurality of data elements into a corresponding set of categories supported by the AI model.
4. The method of claim 1 , wherein applying an AI model of the plurality of AI models to the first dataset comprises:
obtaining embeddings for the plurality of data elements based on the AI model; and
grouping the plurality of data elements into a set of clusters based on the embeddings.
5. The method of claim 4 , wherein grouping the plurality of data elements comprises applying a k-means clustering algorithm to the embeddings.
6. The method of claim 1 , further comprising determining a number of sample data elements to be selected for a respective category of a set of categories associated with a respective AI model.
7. The method of claim 6 , further comprising determining the number of sample data elements based on the set of joint categories obtained from the sets of categories.
8. The method of claim 6 , further comprising determining the number of sample data elements for a category of a set of categories associated with an AI model without considering a category of another AI model.
9. The method of claim 6 , wherein the number of sample data elements selected for the category corresponds to a proportion of data elements for the category in the first dataset.
10. The method of claim 1 , wherein training the first AI model further comprises training the first AI model using the second dataset based on proportions of elements in the first dataset in a respective category of a set of categories associated with a respective AI model.
11. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for facilitating efficient dataset management, the method comprising:
obtaining a first dataset comprising a plurality of data elements for training a first artificial intelligence (AI) model;
determining respective sets of categories of data elements by applying a plurality of AI models to the first dataset, wherein a respective set of categories corresponds to an AI model;
determining a set of joint categories from the sets of categories, wherein a respective joint category corresponds to at least two categories from at least two sets of categories, respectively;
selecting a set of sample data elements associated with a respective category of a respective set of categories by obtaining the set of sample data elements from the set of joint categories;
determining a second dataset based on the selected sample data elements; and
training the first AI model using the second dataset.
12. The non-transitory computer-readable storage medium of claim 11 , wherein the plurality of AI models includes one or more pre-trained classifiers.
13. The non-transitory computer-readable storage medium of claim 11 , wherein applying an AI model of the plurality of AI models to the first dataset comprises categorizing the plurality of data elements into a corresponding set of categories supported by the AI model.
14. The non-transitory computer-readable storage medium of claim 11 , wherein applying an AI model of the plurality of AI models to the first dataset comprises:
obtaining embeddings for the plurality of data elements based on the AI model; and
grouping the plurality of data elements into a set of clusters based on the embeddings.
15. The non-transitory computer-readable storage medium of claim 14 , wherein grouping the plurality of data elements comprises applying a k-means clustering algorithm to the embeddings.
16. The non-transitory computer-readable storage medium of claim 11 , wherein the method further comprises determining a number of sample data elements to be selected for a respective category of a set of categories associated with a respective AI model.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the method further comprises determining the number of sample data elements based on the set of joint categories obtained from the sets of categories.
18. The non-transitory computer-readable storage medium of claim 16 , wherein the method further comprises determining the number of sample data elements for a category of a set of categories associated with an AI model without considering a category of another AI model.
19. The non-transitory computer-readable storage medium of claim 16 , wherein the number of sample data elements selected for the category corresponds to a proportion of data elements for the category in the first dataset.
20. The non-transitory computer-readable storage medium of claim 11 , wherein training the first AI model further comprises training the first AI model using the second dataset based on proportions of elements in the first dataset in a respective category of a set of categories associated with a respective AI model.