Executing machine learning models using transformed datasets
Executing a machine learning model in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: receiving, by a graphical processing unit (‘GPU’) server, a dataset transformed by a storage system that is external to the GPU server; and executing, by the GPU server, one or more machine learning algorithms using the transformed dataset as input.
1 . A method comprising:
applying, by a storage system that is external to a processor, one or more transformations to a dataset to generate a transformed dataset that is usable during execution of a plurality of machine learning algorithms, wherein the one or more transformations applied to the dataset are determined based on an expected input for the plurality of machine learning models,
wherein the processor executes the one or more machine learning algorithms using the transformed dataset as input.
2 . The method of claim 1 further comprising:
identifying, in dependence upon one or more machine learning models to be executed on the processor, one or more transformations to apply to the dataset; and
generating, by the system that is external to the processor and based on the one or more transformations, the transformed dataset.
3 . The method of claim 1 , wherein the system that is external to the processor is a storage system.
4 . The method of claim 1 , wherein the processor is a graphics processing unit (GPU).
5 . The method of claim 1 , wherein receiving the dataset includes receiving the dataset in application memory on the processor.
6 . The method of claim 1 further comprising transmitting, from the system that is external to the processor, the transformed dataset to application memory on the processor.
7 . The method of claim 1 further comprising:
scheduling, by a unified management plane, one or more transformations for the system that is external to the processor to apply to the dataset; and
scheduling, by the unified management plane, execution of one or more machine learning algorithms by the processor, wherein the one or more machine learning algorithms are associated with a machine learning model.
8 . The method of claim 1 further comprising maintaining information describing the dataset, the one or more transformations applied to the dataset, and the transformed dataset.
9 . The method of claim 1 further comprising:
receiving a first request to transmit the transformed dataset to the processor;
transmitting, to the processor, the transformed dataset;
receiving a second request to transmit the transformed dataset to one or more processors; and
transmitting, from the system to the one or more processors without performing an additional transformation of the dataset, the transformed dataset.
10 . An artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, the artificial intelligence infrastructure configured to carry out the steps of:
applying, by a storage system that is external to a processor, one or more transformations to a dataset to generate a transformed dataset that is usable during execution of a plurality of machine learning algorithms, wherein the one or more transformations applied to the dataset are determined based on an expected input for the plurality of machine learning models,
wherein the processor executes the one or more machine learning algorithms using the transformed dataset as input.
11 . The artificial intelligence infrastructure of claim 10 wherein the artificial intelligence infrastructure is further configured to carry out the steps of:
identifying, in dependence upon one or more machine learning models to be executed on the processor, one or more transformations to apply to the dataset; and
generating, by the storage system that is external to the processor and based on the one or more transformations, the transformed dataset.
12 . The artificial intelligence infrastructure of claim 10 wherein receiving the dataset includes receiving the dataset in application memory on the processor.
13 . The artificial intelligence infrastructure of claim 10 wherein the artificial intelligence infrastructure is further configured to carry out the step of transmitting, from the storage system that is external to the processor, the transformed dataset to application memory on the processor.
14 . The artificial intelligence infrastructure of claim 10 wherein the artificial intelligence infrastructure is further configured to carry out the steps of:
scheduling, by a unified management plane, one or more transformations for the system that is external to the processor to apply to the dataset; and
scheduling, by the unified management plane, execution of one or more machine learning algorithms by the processor, wherein the one or more machine learning algorithms are associated with a machine learning model.
15 . The artificial intelligence infrastructure of claim 10 wherein the artificial intelligence infrastructure is further configured to carry out the step of:
maintaining information describing the dataset, the one or more transformations applied to the dataset, and the transformed dataset.
16 . The artificial intelligence infrastructure of claim 10 wherein the artificial intelligence infrastructure is further configured to carry out the steps of:
receiving a first request to transmit the transformed dataset to the processor;
transmitting, to the processor, the transformed dataset;
receiving a second request to transmit the transformed dataset to one or more processors; and
transmitting, from the storage system to the one or more processors without performing an additional transformation of the dataset, the transformed dataset.
17 . An apparatus for data transformation offloading in an artificial intelligence infrastructure that includes one or more storage systems and one or more processors, the apparatus comprising a computer processor, a computer memory operatively coupled to the computer processor, the computer memory having disposed within it computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of:
identifying, in dependence upon one or more machine learning models to be executed on the processors, one or more transformations to apply to a dataset, wherein the one or more transformations applied to the dataset are determined based on an expected input for the one or more machine learning models; and
generating, by the storage system in dependence upon the one or more transformations, a transformed dataset that is usable during execution of a plurality of machine learning algorithms.
18 . The apparatus of claim 17 further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the step of transmitting, from the storage system to the one or more processors, the transformed dataset.
19 . The apparatus of claim 17 wherein transmitting, from the storage system to the one or more processors, the transformed dataset further comprises transmitting the transformed dataset from the one or more storage systems directly to application memory on the processors.
20 . The apparatus of claim 17 further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of:
scheduling, by a unified management plane, one or more transformations for one or more of the storage systems to apply to the dataset; and
scheduling, by the unified management plane, execution of one or more machine learning algorithms associated with the machine learning model by the one or more processor.