Generating a transformed dataset for use by a machine learning model in an artificial intelligence infrastructure
Generating a transformed dataset for use by a machine learning model in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: storing, within one or more storage systems, a transformed dataset generated by applying one or more transformations to a dataset that are identified based on one or more expected input formats of data received as input data by one or more machine learning models to be executed on one or more servers; and transmitting, from the one or more storage systems to the one or more servers without reapplying the one or more transformations on the dataset, the transformed dataset including data in the one or more expected formats of data to be received as input data by the one or more machine learning models.
1. A method comprising:
storing, within one or more storage systems, a transformed dataset generated by applying one or more transformations to a dataset that are identified based on one or more expected input formats of data received as input data by one or more machine learning models to be executed on one or more servers; and
transmitting, from the one or more storage systems to the one or more servers without reapplying the one or more transformations on the dataset, the transformed dataset including data in the one or more expected formats of data to be received as input data by the one or more machine learning models.
2. The method of claim 1 further comprising generating the transformed dataset by applying the transformations to an initial version of the dataset.
3. The method of claim 1 wherein transmitting the transformed dataset further comprises transmitting the transformed dataset from the one or more storage systems directly to application memory on at least one of the servers.
4. The method of claim 3 wherein transmitting the transformed dataset from the one or more storage systems directly to application memory on at least one of the servers further comprises transmitting the transformed data dataset from the one or more storage systems to the server via remote direct memory access (‘RDMA’).
5. The method of claim 1 further comprising executing, by one or more of the servers, one or more machine learning algorithms associated with the machine learning model using the transformed dataset as input.
6. The method of claim 1 further comprising:
scheduling, by a unified management plane, one or more transformations for one or more of the storage systems to apply to the dataset; and
scheduling, by the unified management plane, execution of one or more machine learning algorithms associated with the machine learning model by the one or more servers.
7. The method of claim 1 further comprising providing, by a unified management plane to the one or more servers, information describing the dataset, the one or more transformations applied to the dataset, and the transformed dataset.
8. An artificial intelligence infrastructure configured to carry out steps of:
storing, within one or more storage systems, a transformed dataset generated by applying one or more transformations to a dataset that are identified based on one or more expected input formats of data received as input data by one or more machine learning models to be executed on one or more servers; and
transmitting, from the one or more storage systems to the one or more servers without reapplying the one or more transformations on the dataset, the transformed dataset including data in the one or more expected formats of data to be received as input data by the one or more machine learning models.
9. The artificial intelligence infrastructure of claim 8 wherein the artificial intelligence infrastructure is further configured to carry out the step of generating, by at least one of the storage systems, the transformed dataset by applying the transformations to an initial version of the dataset.
10. The artificial intelligence infrastructure of claim 8 wherein transmitting the transformed dataset further comprises transmitting the transformed dataset from the one or more storage systems directly to application memory on at least one of the servers.
11. The artificial intelligence infrastructure of claim 10 wherein transmitting the transformed dataset from the one or more storage systems directly to application memory on the server further comprises transmitting the transformed data dataset from the one or more storage systems to the server via remote direct memory access (‘RDMA’).
12. The artificial intelligence infrastructure of claim 8 wherein the artificial intelligence infrastructure is further configured to carry out the step of executing, by one or more of the servers, one or more machine learning algorithms associated with the machine learning model using the transformed dataset as input.
13. The artificial intelligence infrastructure of claim 8 wherein the artificial intelligence infrastructure is further configured to carry out the steps of:
scheduling, by a unified management plane, one or more transformations for one or more of the storage systems to apply to the dataset; and
scheduling, by the unified management plane, execution of one or more machine learning algorithms associated with the machine learning model by the one or more servers.
14. The artificial intelligence infrastructure of claim 8 wherein the artificial intelligence infrastructure is further configured to carry out the step of providing, by a unified management plane to the one or more servers, information describing the dataset, the one or more transformations applied to the dataset, and the transformed dataset.
15. An apparatus comprising a computer processor, a computer memory operatively coupled to the computer processor, the computer memory having disposed within it computer program instructions that, when executed by the computer processor, cause the apparatus to carry out steps of:
storing, within one or more storage systems, a transformed dataset generated by applying one or more transformations to a dataset that are identified based on one or more expected input formats of data received as input data by one or more machine learning models to be executed on one or more servers; and
transmitting, from the one or more storage systems to the one or more servers without reapplying the one or more transformations on the dataset, the transformed dataset including data in the one or more expected formats of data to be received as input data by the one or more machine learning models.
16. The apparatus of claim 15 wherein the apparatus further comprises computer program instructions that, when executed by the computer processor, cause the apparatus to carry out step of generating the transformed dataset by applying the transformations to an initial version of the dataset.
17. The apparatus of claim 15 wherein transmitting the transformed dataset further comprises transmitting the transformed dataset from the one or more storage systems directly to application memory on at least one of the servers.
18. The apparatus of claim 15 further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of:
scheduling, by a unified management plane, one or more transformations for one or more of the storage systems to apply to the dataset; and
scheduling, by the unified management plane, execution of one or more machine learning algorithms associated with the machine learning model by the one or more servers.
19. The apparatus of claim 15 further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the step of providing, by a unified management plane to the one or more servers, information describing the dataset, the one or more transformations applied to the dataset, and the transformed dataset.
20. The apparatus of claim 15 further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the step of executing, by one or more of the servers, one or more machine learning algorithms associated with the machine learning model using the transformed dataset as input.