Managing an artificial intelligence infrastructure
Ensuring reproducibility in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: identifying, by a unified management plane, one or more transformations applied to a dataset by the artificial intelligence infrastructure, wherein applying the one or more transformations to the dataset causes the artificial intelligence infrastructure to generate a transformed dataset; storing, within the one or more storage systems, information describing the dataset, the one or more transformations applied to the dataset, and the transformed dataset; identifying, by the unified management plane, one or more machine learning models executed by the artificial intelligence infrastructure using the transformed dataset as input; and storing, within the one or more storage systems, information describing one or more machine learning models executed using the transformed dataset as input.
1. A method comprising:
identifying one or more transformations applied to a dataset, the dataset stored within one or more storage systems that are included in an artificial intelligence infrastructure, wherein the application of the one or more transformations results in a transformed dataset for use in a first machine learning model, and wherein the artificial intelligence infrastructure includes a unified management plane, the one or more storage systems, and one or more graphical processing unit (‘GPU’) servers;
storing, within the one or more storage systems, information describing the dataset, the one or more transformations applied to the dataset, and the transformed dataset;
obtaining, by the unified management plane, identifiers for one or more machine learning models that previously used the transformed dataset as input during execution by the artificial intelligence infrastructure;
storing, within the one or more storage systems, information describing only portions of the one or more machine learning models that differ from the first machine learning model.
2. The method of claim 1 wherein storing, within the one or more storage systems, information describing the dataset, the one or more transformations applied to the dataset, and the transformed dataset further comprises:
generating, by the artificial intelligence infrastructure applying a predetermined hash function to the dataset, the one or more transformations applied to the dataset, and the transformed dataset, a hash value; and
storing, within the one or more storage systems, the hash value.
3. The method of claim 1 wherein storing, within the one or more storage systems, information describing one or more machine learning models executed using the transformed dataset as input further comprises:
generating, by the artificial intelligence infrastructure applying a predetermined hash function to the one or more machine learning models and the transformed dataset, a hash value; and
storing, within the one or more storage systems, the hash value.
4. The method of claim 1 further comprising:
identifying differences between a machine learning model and a machine learning model previously executed by the artificial intelligence infrastructure; and
storing, within the one or more storage systems, only a portion of the machine learning model that differs from the machine learning models previously executed by the artificial intelligence infrastructure.
5. The method of claim 1 further comprising:
determining, by the artificial intelligence infrastructure, whether data related to a previously executed machine learning model should be tiered off of the one or more storage systems; and
responsive to determining that the data related to the previously executed machine learning model should be tiered off of the one or more storage systems:
storing the data related to the previously executed machine learning model in lower-tier storage; and
removing, from the one or more storage systems, the data related to the previously executed machine learning model.
6. The method of claim 1 further comprising identifying, from amongst a plurality of machine learning models, a preferred machine learning model.
7. The method of claim 1 further comprising tracking the improvement of a particular machine learning model over time.
8. An artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, the artificial intelligence infrastructure configured to carry out the steps of:
identifying one or more transformations applied to a dataset, the dataset stored within one or more storage systems that are included in an artificial intelligence infrastructure, wherein the application of the one or more transformations results in a transformed dataset for use in a first machine learning model, and wherein the artificial intelligence infrastructure includes a unified management plane, the one or more storage systems, and one or more graphical processing unit (‘GPU’) servers;
storing, within the one or more storage systems, information describing the dataset, the one or more transformations applied to the dataset, and the transformed dataset;
obtaining, by the unified management plane, identifiers for one or more machine learning models that previously used the transformed dataset as input during execution by the artificial intelligence infrastructure;
storing, within the one or more storage systems, information describing only portions of the one or more machine learning models that differ from the first machine learning model.
9. The artificial intelligence infrastructure of claim 8 wherein storing, within the one or more storage systems, information describing the dataset, the one or more transformations applied to the dataset, and the transformed dataset further comprises:
generating, by the artificial intelligence infrastructure applying a predetermined hash function to the one or more transformations applied to the dataset and the transformed dataset, a hash value; and
storing, within the one or more storage systems, the hash value.
10. The artificial intelligence infrastructure of claim 8 wherein storing, within the one or more storage systems, information describing one or more machine learning models executed using the transformed dataset as input further comprises:
generating, by the artificial intelligence infrastructure applying a predetermined hash function to the one or more machine learning models, a hash value; and
storing, within the one or more storage systems, the hash value.
11. The artificial intelligence infrastructure of claim 8 wherein the artificial intelligence infrastructure is further configured to carry out the steps of:
identifying, by the unified management plane, differences between a machine learning model and a machine learning model previously executed by the artificial intelligence infrastructure; and
storing, within the one or more storage systems, only a portion of the machine learning model that differs from the machine learning models previously executed by the artificial intelligence infrastructure.
12. The artificial intelligence infrastructure of claim 8 wherein the artificial intelligence infrastructure is further configured to carry out the steps of:
determining, by the artificial intelligence infrastructure, whether data related to a previously executed machine learning model should be tiered off of the one or more storage systems; and
responsive to determining that the data related to the previously executed machine learning model should be tiered off of the one or more storage systems:
storing the data related to the previously executed machine learning model in lower-tier storage; and
removing, from the one or more storage systems, the data related to the previously executed machine learning model.
13. The artificial intelligence infrastructure of claim 8 wherein the artificial intelligence infrastructure is further configured to carry out the step of identifying, from amongst a plurality of machine learning models, a preferred machine learning model.
14. The artificial intelligence infrastructure of claim 8 wherein the artificial intelligence infrastructure is further configured to carry out the step of tracking the improvement of a particular machine learning model over time.
15. An apparatus comprising a computer processor, a computer memory operatively coupled to the computer processor, the computer memory having disposed within it computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of:
identifying one or more transformations applied to a dataset, the dataset stored within one or more storage systems that are included in an artificial intelligence infrastructure, wherein the application of the one or more transformations results in a transformed dataset for use in a first machine learning model, and wherein the artificial intelligence infrastructure includes a unified management plane, the one or more storage systems, and one or more graphical processing unit (‘GPU’) servers;
storing, within the one or more storage systems, information describing the dataset, the one or more transformations applied to the dataset, and the transformed dataset;
obtaining, by the unified management plane, identifiers for one or more machine learning models that previously used the transformed dataset as input during execution by the artificial intelligence infrastructure;
storing, within the one or more storage systems, information describing only portions of the one or more machine learning models that differ from the first machine learning model.
16. The apparatus of claim 15 further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of:
identifying, by the unified management plane, differences between a machine learning model and a machine learning model previously executed by the artificial intelligence infrastructure; and
storing, within the one or more storage systems, only a portion of the machine learning model that differs from the machine learning models previously executed by the artificial intelligence infrastructure.
17. The apparatus of claim 15 further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of:
determining, by the artificial intelligence infrastructure, whether data related to a previously executed machine learning model should be tiered off of the one or more storage systems; and
responsive to determining that the data related to the previously executed machine learning model should be tiered off of the one or more storage systems:
storing the data related to the previously executed machine learning model in lower-tier storage; and
removing, from the one or more storage systems, the data related to the previously executed machine learning model.
18. The apparatus of claim 15 further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the step of identifying, from amongst a plurality of machine learning models, a preferred machine learning model.
19. The apparatus of claim 15 further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the step of tracking the improvement of a particular machine learning model over time.
20. The apparatus of claim 15 further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of:
generating, by the artificial intelligence infrastructure applying a predetermined hash function to the dataset, and the one or more transformations applied to the dataset, a hash value; and
storing, within the one or more storage systems, the hash value.