Using consistency metadata for filtering of machine learning data across jobs
Consistency metadata, including a parameter for a pseudo-random number source, are determined for training-and-evaluation iterations of a machine learning model. Using the metadata, a first training set comprising records of at least a first chunk is identified from a plurality of chunks of a data set. The first training set is used to train a machine learning model during a first training-and-evaluation iteration. A first test set comprising records of at least a second chunk is identified using the metadata, and is used to evaluate the model during the first training-and-evaluation iteration.
1 . A computer-implemented method, comprising:
scheduling, at a first set of servers by a machine learning service of a cloud computing environment in response to one or more programmatic requests from a client, a training job for a particular machine learning model, wherein a parameter of the one or more programmatic requests comprises consistency metadata to be used to ensure that a test set used for the particular machine learning model does not overlap with a training set of the machine learning model, wherein the first set of servers has access to a first pseudo-random number source, and wherein one or more pseudo-random numbers obtained from the first pseudo-random number source are used at the first set of servers to select, as part of the training job, a training set for the particular machine learning model from a data set;
scheduling, at a second set of servers by the machine learning service in response to the one or more programmatic requests, a test job for the particular machine learning model, wherein the second set of servers has access to a second pseudo-random number source; and
selecting, at the second set of servers as part of the test job, a test set for the particular machine learning model from the data set, wherein the test set is selected using one or more pseudo-random numbers obtained from the second pseudo-random number source after synchronizing, using at least the consistency metadata, a state of the second pseudo-random number source with a state of the first pseudo-random number source.
2 . The computer-implemented method as recited in claim 1 , wherein the consistency metadata comprises a seed value.
3 . The computer-implemented method as recited in claim 1 , wherein synchronizing the state of the second pseudo-random number source comprises:
obtaining a plurality of pseudo-random numbers from the second pseudo-random number source.
4 . The computer-implemented method as recited in claim 1 , wherein another parameter of the one or more parameters indicates a ratio of a size of the training set to a size of the test set, and wherein the test set is selected in accordance with the ratio.
5 . The computer-implemented method as recited in claim 1 , wherein another parameter of the one or more parameters indicates that a plurality of training-and-evaluation iterations is to be performed for the machine learning model, wherein the training set is utilized for a particular training-and-evaluation iteration of the plurality of training-and-evaluation iterations, and wherein the test set is utilized for the particular training-and-evaluation iteration.
6 . The computer-implemented method as recited in claim 1 , wherein the one or more parameters indicate that the data set is distributed among a plurality of data sources, wherein the one or more programmatic requests are received via a data-source-agnostic interface of the machine learning service, and wherein selecting the training set comprises:
extracting at least a first record of a plurality of records of the training set from a first data source of the plurality of data sources; and
extracting at least a second record of the plurality of records from a second data source of the plurality of data sources.
7 . The computer-implemented method as recited in claim 1 , further comprising:
receiving, at the machine learning service, a first request to shuffle another data set;
saving, at the machine learning service, additional consistency metadata which includes state information associated with one or more other pseudo-random numbers used to shuffle the other data set; and
re-obtaining, at the machine learning service in response to a second request to shuffle the other data set, results of a first shuffle operation performed in response to the first shuffle request, wherein said re-obtaining comprises using the additional consistency metadata.
8 . A system, comprising:
one or more computing devices;
wherein the one or more computing devices include instructions that upon execution on or across the one or more computing devices:
schedule, at a first set of servers by a machine learning service of a cloud computing environment in response to one or more programmatic requests from a client, a training job for a particular machine learning model, wherein a parameter of the one or more programmatic requests comprises consistency metadata to be used to ensure that a test set used for the particular machine learning model does not overlap with a training set of the machine learning model, wherein the first set of servers has access to a first pseudo-random number source, and wherein one or more pseudo-random numbers obtained from the first pseudo-random number source are used at the first set of servers to select, as part of the training job, a training set for the particular machine learning model from a data set;
schedule, at a second set of servers by the machine learning service in response to the one or more programmatic requests, a test job for the particular machine learning model, wherein the second set of servers has access to a second pseudo-random number source; and
select, at the second set of servers as part of the test job, a test set for the particular machine learning model from the data set, wherein the test set is selected using one or more pseudo-random numbers obtained from the second pseudo-random number source after synchronizing, using at least the consistency metadata, a state of the second pseudo-random number source with a state of the first pseudo-random number source.
9 . The system as recited in claim 8 , wherein the consistency metadata comprises a seed value.
10 . The system as recited in claim 8 , wherein to synchronize the state of the second pseudo-random number source, the one or more computing devices include further instructions that upon execution on or across the one or more computing devices:
obtain a plurality of pseudo-random numbers from the second pseudo-random number source.
11 . The system as recited in claim 8 , wherein another parameter of the one or more parameters indicates a ratio of a size of the training set to a size of the test set, and wherein the test set is selected in accordance with the ratio.
12 . The system as recited in claim 8 , wherein another parameter of the one or more parameters indicates that a plurality of training-and-evaluation iterations is to be performed for the machine learning model, wherein the training set is utilized for a particular training-and-evaluation iteration of the plurality of training-and-evaluation iterations, and wherein the test set is utilized for the particular training-and-evaluation iteration.
13 . The system as recited in claim 8 , wherein the one or more parameters indicate that the data set is distributed among a plurality of data sources, wherein the one or more programmatic requests are received via a data-source-agnostic interface of the machine learning service, and wherein to select the training set, the one or more computing devices include further instructions that upon execution on or across the one or more computing devices:
extract at least a first record of a plurality of records of the training set from a first data source of the plurality of data sources; and
extract at least a second record of the plurality of records from a second data source of the plurality of data sources.
14 . The system as recited in claim 8 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more computing devices:
receive, at the machine learning service, a first request to shuffle another data set;
save, at the machine learning service, additional consistency metadata which includes state information associated with one or more other pseudo-random numbers used to shuffle the other data set; and
re-obtain, at the machine learning service in response to a second request to shuffle the other data set, results of a first shuffle operation performed in response to the first shuffle request, wherein said re-obtaining comprises using the additional consistency metadata.
15 . One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors:
schedule, at a first set of servers by a machine learning service of a cloud computing environment in response to one or more programmatic requests from a client, a training job for a particular machine learning model, wherein a parameter of the one or more programmatic requests comprises consistency metadata to be used to ensure that a test set used for the particular machine learning model does not overlap with a training set of the machine learning model, wherein the first set of servers has access to a first pseudo-random number source, and wherein one or more pseudo-random numbers obtained from the first pseudo-random number source are used at the first set of servers to select, as part of the training job, a training set for the particular machine learning model from a data set;
schedule, at a second set of servers by the machine learning service in response to the one or more programmatic requests, a test job for the particular machine learning model, wherein the second set of servers has access to a second pseudo-random number source; and
select, at the second set of servers as part of the test job, a test set for the particular machine learning model from the data set, wherein the test set is selected using one or more pseudo-random numbers obtained from the second pseudo-random number source after synchronizing, using at least the consistency metadata, a state of the second pseudo-random number source with a state of the first pseudo-random number source.
16 . The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the consistency metadata comprises a seed value.
17 . The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein to synchronize the state of the second pseudo-random number source, the one or more non-transitory computer-accessible storage media store further program instructions that when executed on or across the one or more processors:
obtain a plurality of pseudo-random numbers from the second pseudo-random number source.
18 . The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein another parameter of the one or more parameters indicates a ratio of a size of the training set to a size of the test set, and wherein the test set is selected in accordance with the ratio.
19 . The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein another parameter of the one or more parameters indicates that a plurality of training-and-evaluation iterations is to be performed for the machine learning model, wherein the training set is utilized for a particular training-and-evaluation iteration of the plurality of training-and-evaluation iterations, and wherein the test set is utilized for the particular training-and-evaluation iteration.
20 . The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the one or more parameters indicate that the data set is distributed among a plurality of data sources, wherein the one or more programmatic requests are received via a data-source-agnostic interface of the machine learning service, and wherein to select the training set, the one or more non-transitory computer-accessible storage media store further program instructions that when executed on or across the one or more processors:
extract at least a first record of a plurality of records of the training set from a first data source of the plurality of data sources; and
extract at least a second record of the plurality of records from a second data source of the plurality of data sources.