Creating Batches Of Training Data For Machine Learning Workflows
Creating batches of training data for machine learning workflows, including: selecting, by an artificial intelligence and machine learning infrastructure system in accordance with a batch building policy, a subset of data objects stored in a data storage system, wherein the batch building policy describes parameters for selecting data types; and providing, to a deep learning computing system by the artificial intelligence and machine learning infrastructure system, one or more data objects that include the subset of data objects selected based on the parameters for selecting data types described by the batch building policy.
1 . A method comprising:
selecting, by an artificial intelligence and machine learning infrastructure system in accordance with a batch building policy, a subset of data objects stored in a data storage system, wherein the batch building policy describes parameters for selecting data types; and
providing, to a deep learning computing system by the artificial intelligence and machine learning infrastructure system, one or more data objects that include the subset of data objects selected based on the parameters for selecting data types described by the batch building policy.
2 . The method of claim 1 , wherein the batch building policy specifies a collection of data objects selected from among multiple data objects stored among one or more directories.
3 . The method of claim 2 , wherein selecting the subset of data objects includes generating parallel respective remote procedure calls for respective data objects among the one or more directories.
4 . The method of claim 2 , wherein, based on respective types of data objects being stored within respective directories of the one or more directories, the batch building policy specifies a collection of data objects such that data objects are selected from a maximum quantity of different directories.
5 . The method of claim 2 , wherein the batch building policy specifies a subset of types of data objects from among multiple types of data objects stored among the one or more directories.
6 . The method of claim 2 , wherein the batch building policy specifies a subset of types of data objects from among multiple types of data objects stored among the one or more directories.
7 . The method of claim 2 , wherein the batch building policy specifies a collection of data objects selected according to distribution of different data object types among multiple data types of the multiple data objects stored among the one or more directories.
8 . The method of claim 1 , wherein the batch building policy specifies a collection of data objects randomly selected from among multiple data objects stored among one or more directories.
9 . The method of claim 1 , wherein the batch building policy specifies a collection of data objects selected from among multiple data objects such that the collection of data objects includes a balanced selection of different types of data objects.
10 . The method of claim 1 , wherein the batch building policy specifies a collection of data objects selected from among multiple data objects such that the collection of data objects includes a shuffled selection of different types of data objects.
11 . An artificial intelligence and machine learning infrastructure system comprising:
one or more storage systems comprising, respectively, one or more storage devices; and
one or more graphical processing units, wherein the graphical processing units are configured to communicate with the one or more storage systems over a communication fabric;
wherein the artificial intelligence and machine learning infrastructure system is configured to:
select, in accordance with a batch building policy, a subset of data objects stored in a data storage system, wherein the batch building policy describes parameters for selecting data types; and
provide, to a deep learning computing system, one or more data objects that include the subset of data objects selected based on the parameters for selecting data types described by the batch building policy.
12 . The artificial intelligence and machine learning infrastructure system of claim 11 , wherein the batch building policy specifies a collection of data objects selected from among multiple data objects stored among one or more directories.
13 . The artificial intelligence and machine learning infrastructure system of claim 12 , wherein selecting the subset of data objects includes generating parallel respective remote procedure calls for respective data objects among the one or more directories.
14 . The artificial intelligence and machine learning infrastructure system of claim 12 , wherein, based on respective types of data objects being stored within respective directories of the one or more directories, the batch building policy specifies a collection of data objects such that data objects are selected from a maximum quantity of different directories.
15 . The artificial intelligence and machine learning infrastructure system of claim 12 , wherein the batch building policy specifies a subset of types of data objects from among multiple types of data objects stored among the one or more directories.
16 . The artificial intelligence and machine learning infrastructure system of claim 12 , wherein the batch building policy specifies a subset of types of data objects from among multiple types of data objects stored among the one or more directories.
17 . The artificial intelligence and machine learning infrastructure system of claim 12 , wherein the batch building policy specifies a collection of data objects selected according to distribution of different data object types among multiple data types of the multiple data objects stored among the one or more directories.
18 . The artificial intelligence and machine learning infrastructure system of claim 11 , wherein the batch building policy specifies a collection of data objects randomly selected from among multiple data objects stored among one or more directories.
19 . The artificial intelligence and machine learning infrastructure system of claim 11 , wherein the batch building policy specifies a collection of data objects selected from among multiple data objects such that the collection of data objects includes a balanced selection of different types of data objects.
20 . The artificial intelligence and machine learning infrastructure system of claim 11 , wherein the batch building policy specifies a collection of data objects selected from among multiple data objects such that the collection of data objects includes a shuffled selection of different types of data objects.