Batch building for deep learning training workloads
Batch building for artificial intelligence workflows, including: issuing, responsive to a request for a batch of data objects, requests to a data repository for multiple data objects stored among one or more directories; selecting, in accordance with a batch building policy, a subset of data objects based on one or more responses to the requests; and providing, to the artificial intelligence workflow, a batch of data objects that includes the subset of data objects.
1. A method of batch building for artificial intelligence workflows, the method comprising:
issuing, by an artificial intelligence and machine learning infrastructure system responsive to a request from a deep learning computing system for a batch of data objects, requests to a data storage system for multiple data objects stored among one or more directories;
selecting, by the artificial intelligence and machine learning infrastructure system in accordance with a batch building policy, a subset of data objects stored in the data storage system based on one or more responses to the requests, wherein the batch building policy describes a sequence of data types, and wherein the one or more responses comprise characteristics of files within a file system of the data storage system; and
providing, to the deep learning computing system by the artificial intelligence and machine learning infrastructure system responsive to the request for the batch of data objects, one or more data objects that include the subset of data objects in the sequence described by the batch building policy.
2. The method of claim 1 , wherein the request for the batch of data objects is received from an artificial intelligence workflow.
3. The method of claim 2 , wherein the artificial intelligence workflow executes within an artificial intelligence and machine learning infrastructure.
4. The method of claim 1 , wherein the batch building policy specifies a collection of data objects randomly selected from among the multiple data objects stored among the one or more directories.
5. The method of claim 1 , wherein the batch building policy specifies a collection of data objects selected from among the multiple data objects such that the batch of data objects includes a balanced selection of different types of data objects.
6. The method of claim 1 , wherein the batch building policy specifies a collection of data objects selected from among the multiple data objects such that the batch of data objects includes a shuffled selection of different types of data objects.
7. The method of claim 1 , wherein, based on respective types of data objects being stored within respective directories of the one or more directories, the batch building policy specifies a collection of data objects such that data objects are selected from a maximum quantity of different directories.
8. The method of claim 1 , wherein the batch building policy specifies a subset of types of data objects from among multiple types of data objects stored among the one or more directories.
9. The method of claim 1 , wherein the batch building policy specifies a collection of data objects selected according to distribution of different data object types among multiple data types of the multiple data objects stored among the one or more directories.
10. The method of claim 1 , wherein issuing the requests includes generating parallel respective remote procedure calls for respective data objects among the one or more directories.
11. An artificial intelligence and machine learning infrastructure system comprising:
one or more storage systems comprising, respectively, one or more storage devices; and
one or more graphical processing units, wherein the graphical processing units are configured to communicate with the one or more storage systems over a communication fabric;
wherein the artificial intelligence and machine learning infrastructure system is configured to:
issue, responsive to a request for a batch of data objects from a deep learning computing system, requests to a data storage system for multiple data objects stored among one or more directories;
select, in accordance with a batch building policy, a subset of data objects stored in the data storage system based on one or more responses to the requests, wherein the batch building policy describes a sequence of data types, and wherein the one or more responses comprise characteristics of files within a file system of the data storage system; and
provide, to the deep learning computing system responsive to the request for the batch of data objects, one or more data objects that include the subset of data objects in the sequence described by the batch building policy.
12. The artificial intelligence and machine learning infrastructure system of claim 11 , wherein the request for the batch of data objects is received from an artificial intelligence workflow.
13. The artificial intelligence and machine learning infrastructure system of claim 12 , wherein the artificial intelligence workflow executes within an artificial intelligence and machine learning infrastructure.
14. The artificial intelligence and machine learning infrastructure system of claim 11 , wherein the batch building policy specifies a collection of data objects randomly selected from among the multiple data objects stored among the one or more directories.
15. The artificial intelligence and machine learning infrastructure system of claim 11 , wherein the batch building policy specifies a collection of data objects selected from among the multiple data objects such that the batch of data objects includes a balanced selection of different types of data objects.
16. The artificial intelligence and machine learning infrastructure system of claim 11 , wherein the batch building policy specifies a collection of data objects selected from among the multiple data objects such that the batch of data objects includes a shuffled selection of different types of data objects.
17. The artificial intelligence and machine learning infrastructure system of claim 11 , wherein, based on respective types of data objects being stored within respective directories of the one or more directories, the batch building policy specifies a collection of data objects such that data objects are selected from a maximum quantity of different directories.
18. The artificial intelligence and machine learning infrastructure system of claim 11 , wherein the batch building policy specifies a subset of types of data objects from among multiple types of data objects stored among the one or more directories.
19. The artificial intelligence and machine learning infrastructure system of claim 11 , wherein the batch building policy specifies a collection of data objects selected according to distribution of different data object types among multiple data types of the multiple data objects stored among the one or more directories.
20. The artificial intelligence and machine learning infrastructure system of claim 11 , wherein issuing the requests includes generating parallel respective remote procedure calls for respective data objects among the one or more directories.