Machine learning data set management for distribution to machine learning applications
Centralized data set management techniques are implemented for data sets for machine learning applications. New data sets may be created from a search of managed data set labels. Data objects of the managed data sets that are found in the search are then used to generate a new data set. The new data set is then made available for machine learning applications to use.
1 . A system, comprising:
at least one processor; and
a memory, storing program instructions that when executed by the at least one processor, cause the at least one processor to:
store, by a machine learning data set management system, a plurality of data sets respectively comprising a plurality of data objects with respective labels and respective metadata describing the one or more data sets;
receive, via an interface of the machine learning data set management system, a request to search the plurality of data sets according to one or more search criteria;
provide, via the interface of the machine learning data set management system, one or more data sets of the plurality of data sets identified based, at least in part on:
respective access rights of the plurality of data sets that allow sharing of the one or more data sets with an identity associated with the request to search the plurality of data sets; and
one or more search criteria;
receive, via the interface of the machine learning data set management system, a request to create a new data set to be made available for access by one or more machine learning applications, the new data set to be a subset of the one or more data sets, wherein the request specifies one or more labels used to identify one or more data objects from the plurality of data objects in the one or more data sets to create the new data set;
responsive to the request:
determine, by the machine learning data set management system, that the one or more data sets allow creation of the new data set based at least in part on the respective access rights and the identity:
search, by the machine learning data set management system, the respective metadata describing the one or more data sets and the respective labels of the plurality of data objects included in the one or more data sets according to the specified one or more labels, wherein to search, the machine learning data set management system:
compares the respective labels of the plurality of data objects to identify the one or more data objects that match at least one of the specified one or more labels to include in the new data set;
create, by the machine learning data set management system, the new data set that includes the identified one or more data objects from the one or more data sets; and
after creation of the new data set, make, by the machine learning data set management system, the new data set available for access by the one or more machine learning applications.
2 . The system of claim 1 , wherein the memory stores further program instructions that when executed by the at least one processor, cause the at least one processor further to update lineage data for the new data set after determining that the new data set has been accessed by one of the one or more machine learning applications.
3 . The system of claim 1 , wherein the memory stores further program instructions that when executed by the at least one processor, cause the at least one processor further to:
receive, via the interface of the machine learning data set management system, a request to add the one or more data sets to be managed by the machine learning data set management system;
obtain, by the machine learning data set management system, the one or more data sets;
determine and perform, by the machine learning data set management system, one or more import tasks to process the data set for management by the machine learning data set management system;
create and store, by the machine learning data set management system, metadata for managing the processed data set.
4 . The system of claim 1 , wherein the machine learning data set management system is a service offered by a provider network, wherein the one or more data sets are stored as part of a storage service of the provider network, wherein the new data set is stored as part of the storage service of the provider network, and wherein the one or more machine learning applications are one or more machine learning services of the provider network.
5 . A method, comprising:
receiving, via an interface of the machine learning data set management system, a request to search the plurality of data sets according to one or more search criteria;
providing, via the interface of the machine learning data set management system, one or more data sets of the plurality of data sets identified based, at least in part on:
respective access rights of the plurality of data sets that allow sharing of the one or more data sets with an identity associated with the request to search the plurality of data sets; and
one or more search criteria;
receiving, via the interface of a machine learning data set management system, a request to create a new data set to be made available for access by one or more machine learning applications, the new data set to be a subset of the one or more data sets, wherein the request specifies one or more labels used to identify one or more data objects from a plurality of data objects stored in the one or more data sets to create the new data set;
responsive to the request:
determining, by the machine learning data set management system, that the one or more data sets allow creation of the new data set based at least in part on the respective access rights and the identity;
searching, by the machine learning data set management system, respective labels of the plurality of data objects included in the one or more data sets according to the specified one or more labels, wherein the searching comprises comparing the respective labels of the plurality of data objects to identify the one or more data objects that match at least one of the specified one or more labels to include in the new data set;
creating, by the machine learning data set management system, the new data set that includes the identified one or more data objects from the one or more data sets; and
after creating the new data set, making, by the machine learning data set management system, the new data set available for access by the one or more machine learning applications.
6 . The method of claim 5 , further comprising updating lineage data for the new data set after determining that the new data set has been accessed by one of the one or more machine learning applications.
7 . The method of claim 5 , further comprising granting access to the new data set to one or more additional users of the machine learning data set management system, wherein the new data set was created on behalf of a first user of the machine learning data set management system.
8 . The method of claim 5 , further comprising:
receiving, via the interface of the machine learning data set management system, a request to add the one or more data sets to be managed by the machine learning data set management system;
obtaining, by the machine learning data set management system, the one or more data sets;
determining and performing, by the machine learning data set management system, one or more import tasks to process the data set for management by the machine learning data set management system;
storing, by the machine learning data set management system, the processed one or more data sets; and
creating and storing, by the machine learning data set management system, metadata for managing the processed data set.
9 . The method of claim 8 , further comprising performing one or more statistical analyses on the one or more data sets to determine at least some of the metadata for the one or more data sets.
10 . The method of claim 8 , further comprising performing one or more machine learning analyses on the one or more data sets to modify the one or more data sets.
11 . The method of claim 5 , wherein the one or more search criteria is a query that specifies at least one label for data objects.
12 . The method of claim 5 , wherein one of the one or more machine learning applications is a data labeling application, wherein the data labeling application is provided access to the new data set to perform a data labeling job to add one or more labels to different ones of the data objects in the new data set.
13 . The method of claim 5 , further comprising displaying, via the interface of the machine learning data set management system, lineage data for the new data set responsive to a request for lineage data for the new data set received via the interface of the machine learning data set management system.
14 . One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement:
receiving, via an interface of the machine learning data set management system, a request to search a plurality of data sets according to one or more search criteria;
providing, via the interface of the machine learning data set management system, one or more data sets of the plurality of data sets identified based, at least in part on:
respective access rights of the plurality of data sets that allow sharing of the one or more data sets with an identity associated with the request to search the plurality of data sets; and
one or more search criteria;
receiving, via the interface of a machine learning data set management system, a request to create a new data set to be made available for access by one or more machine learning applications, the new data set to be a subset of the one or more data sets, wherein the request specifies one or more labels used to identify one or more data objects from the plurality of data objects in the one or more data sets to create the new data set;
responsive to the request:
determining, by the machine learning data set management system, that the one or more data sets allow creation of the new data set based at least in part on the respective access rights and the identity:
searching, by the machine learning data set management system, respective labels of the plurality of data objects included in the one or more data sets according to the specified one or more labels, wherein the searching comprises comparing the respective labels of the plurality of data objects to identify the one or more data objects that match at least one of the specified one or more labels to include in the new data set;
creating, by the machine learning data set management system, the new data set that includes the identified one or more data objects from the one or more data sets; and
after creating the new data set, making, by the machine learning data set management system, the new data set available for access by the one or more machine learning applications.
15 . The one or more non-transitory, computer-readable storage media of claim 14 , wherein the new data set is provided to a first machine learning application and a second machine learning application, and wherein the non-transitory, computer- readable storage media store further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further providing a performance comparison of the first machine learning application and the second machine learning application with respect to the new data set via the interface of the machine learning data set management system.
16 . The one or more non-transitory, computer-readable storage media of claim 14 , storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement:
receiving, via the interface of the machine learning data set management system, a request to add the one or more data sets to be managed by the machine learning data set management system;
obtaining, by the machine learning data set management system, the one or more data sets;
determining and performing, by the machine learning data set management system, one or more import tasks to process the data set for management by the machine learning data set management system;
storing, by the machine learning data set management system, the processed one or more data sets; and
creating and storing, by the machine learning data set management system, metadata for managing the processed data set.
17 . The one or more non-transitory, computer-readable storage media of claim 16 , storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement performing one or more machine learning analyses on the one or more data sets to modify the one or more data sets as part of performing the import tasks determined for the request to add the one or more data sets to be managed by the machine learning data set management system.
18 . The one or more non-transitory, computer-readable storage media of claim 16 , storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement anonymizing the one or more data sets as part of performing the import tasks determined for the request to add the one or more data sets to be managed by the machine learning data set management system.
19 . The one or more non-transitory, computer-readable storage media of claim 14 , storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement generating one or more synthetic data objects to include in the new data set.
20 . The one or more non-transitory, computer-readable storage media of claim 14 , wherein the machine learning data set management system is a service offered by a provider network, wherein the one or more data sets are stored as part of a storage service of the provider network, wherein the new data set is stored as part of the storage service of the provider network, and wherein the one or more machine learning applications are one or more machine learning services of the provider network.