Training artificial intelligence workflows
Nonsequential readahead for deep learning training that includes: receiving an indication of a list of batch storage locations for a batch of data objects; prefetching, for each storage location in the list of batch storage locations, storage content corresponding to the batch of data objects; and storing the storage content corresponding to the batch of data objects within a cache accessible to an artificial intelligence workflow.
1. A method comprising:
prefetching, using an indication of a list of batch storage locations for a batch of data objects, storage content corresponding to the batch of data objects; and
training an artificial intelligence workflow using a non-sequential readahead of the storage content, wherein randomization is implemented on the list of batch storage objects in the non-sequential readahead to perform a randomized prefetch of data objects from the batch of data objects from the list of batch storage locations.
2. The method of claim 1 , further comprising:
transforming, by a storage system, the batch of data objects to generate a transformed batch of data objects; and
training the artificial intelligence workflow using the transformed batch of data objects.
3. The method of claim 1 , wherein the indication of the list specifies a beginning of the list of batch storage locations.
4. The method of claim 1 , wherein the indication of the list specifies a quantity of memory locations from which to obtain the batch of data objects.
5. The method of claim 1 , wherein the indication of the list specifies a range of memory locations from which to obtain the batch of data objects.
6. The method of claim 1 , wherein prefetching includes using a hardware level controller that accesses the list of batch storage locations.
7. The method of claim 1 , wherein the artificial intelligence workflow executes within an artificial intelligence and machine learning infrastructure.
8. An artificial intelligence and machine learning infrastructure comprising:
one or more storage systems comprising, respectively, one or more storage devices; and
one or more graphical processing units, wherein the graphical processing units are configured to communicate with the one or more storage systems over a communication fabric;
wherein an artificial intelligence workflow executing on at least one of the one or more graphical processing units is configured to:
prefetch, using an indication of a list of batch storage locations for a batch of data objects, storage content corresponding to the batch of data objects; and
train an artificial intelligence workflow using a non-sequential readahead of the storage content, wherein randomization is implemented on the list of batch storage objects in the non-sequential readahead to perform a randomized prefetch of data objects from the batch of data objects from the list of batch storage locations.
9. The artificial intelligence and machine learning infrastructure system of claim 8 , wherein the artificial intelligence workflow is configured to:
transform, by the storage system, the batch of data objects to generate a transformed batch of data objects; and
train the artificial intelligence workflow using the transformed batch of data objects.
10. The artificial intelligence and machine learning infrastructure system of claim 8 , wherein the indication of the list specifies a beginning of the list of batch storage locations.
11. The artificial intelligence and machine learning infrastructure system of claim 8 , wherein the indication of the list specifies a quantity of memory locations from which to obtain the batch of data objects.
12. The artificial intelligence and machine learning infrastructure system of claim 8 , wherein the indication of the list specifies a range of memory locations from which to obtain the batch of data objects.
13. The artificial intelligence and machine learning infrastructure system of claim 8 , wherein prefetching includes using a hardware level controller that accesses the list of batch storage locations.
14. The artificial intelligence and machine learning infrastructure system of claim 8 , wherein the artificial intelligence workflow executes within an artificial intelligence and machine learning infrastructure.
15. An apparatus for nonsequential readahead, the apparatus comprising a computer processor, a computer memory operatively coupled to the computer processor, the computer memory having disposed within it computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of:
prefetching, using an indication of a list of batch storage locations for a batch of data objects, storage content corresponding to the batch of data objects; and
training an artificial intelligence workflow using a non-sequential readahead of the storage content, wherein randomization is implemented on the list of batch storage objects in the non-sequential readahead to perform a randomized prefetch of data objects from the batch of data objects from the list of batch storage locations.
16. The apparatus of claim 15 , wherein the instructions further cause the apparatus to carry out the steps of:
transforming, by a storage system, the batch of data objects to generate a transformed batch of data objects; and
training the artificial intelligence workflow using the transformed batch of data objects.
17. The apparatus of claim 15 , wherein the indication of the list specifies a beginning of the list of batch storage locations.
18. The apparatus of claim 17 , wherein the indication of the list specifies a quantity of memory locations from which to obtain the batch of data objects.
19. The apparatus of claim 15 , wherein the indication of the list specifies a range of memory locations from which to obtain the batch of data objects.
20. The apparatus of claim 15 , wherein prefetching includes using a hardware level controller that accesses the list of batch storage locations.