Unified machine learning feature data pipeline
A unified system with a machine learning feature data pipeline that can be shared among various product areas or teams of an electronic platform is described. A set of features can be fetched from multiple feature sources. The set of features can be combined with browsing event data to generate combined data. The combined data can be sampled to generate sampled data. The sampled data can be presented in a format having a structure that is agnostic to a feature source from which the set of features was fetched. The sampled data can be joined with old features by a backfilling process to generate training data designed to train one or more machine learning models. Related methods, apparatuses, articles of manufacture, and computer program products are also described.
1 . A method comprising:
fetching, by one or more data processing apparatuses and over a network, a set of features from a plurality of feature sources;
constructing, by the one or more data processing apparatuses and using the set of features, a set of training data having a structure that is agnostic to a source from which the features were received;
detecting, by the one or more data processing apparatuses, that (i) a change has occurred to a given feature, among the set of features that were used to construct the set of training data, that has not been logged in the set of features or (ii) a new feature exists that has not been logged in the set of features;
regenerating, by the one or more data processing apparatuses and prior to logging of the change that occurred to the given feature or the new feature that exists, the set of training data by joining the set of training data with the detected (i) change that occurred to the given feature or (ii) the new feature that exists, wherein the joining is performed based on a key of the training data;
training, prior to logging the detected change to the given feature or the new feature that exists, a given machine learning model using the regenerated set of training data; and
logging the change that occurred to the given feature or the new feature.
2 . The method of claim 1 , wherein the constructing of the set of training data comprises:
combining, by the one or more data processing apparatuses, the set of features with a browsing event data to generate combined data;
sampling, by the one or more data processing apparatuses, the combined data to generate sampled data, wherein the sampled data is presented in a format having a structure that is agnostic to a feature source from which the set of features was fetched; and
joining, by the one or more data processing apparatuses, the sampled data with old features by a back-filling process to generate training data designed to train one or more machine learning models.
3 . The method of claim 2 , wherein the one or more machine learning models are configured to be used to make predictions for respective one or more product areas of an electronic platform.
4 . The method of claim 2 , wherein the sampling comprises stratified sampling that applies weights to the set of features within the combined data, wherein the weights vary based on one or more of recency or frequency of browsing activity within the browsing event data.
5 . The method of claim 1 , wherein the set of features include (i) batch features and (ii) streaming features that are updated more often than the batch features.
6 . The method of claim 5 , wherein the batch features are stored in a first set of columns, wherein the streaming features are stored in a second set of columns, wherein the second set of columns are updated more often than the first set of columns.
7 . A system comprising:
at least one programmable processor; and
a machine-readable medium storing instructions that, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
fetching, over a network, a set of features from a plurality of feature sources;
constructing, using the set of features, a set of training data having a structure that is agnostic to a source from which the features were received;
detecting that (i) a change has occurred to a given feature, among the set of features that were used to construct the set of training data, that has not been logged in the set of features or (ii) a new feature exists that has not been logged in the set of features;
regenerating, prior to logging of the change that occurred to the given feature or the new feature that exists, the set of training data by joining the set of training data with the detected (i) change that occurred to the given feature or (ii) the new feature that exists, wherein the joining is performed based on a key of the training data;
training, prior to logging the detected change to the given feature or the new feature that exists, a given machine learning model using the regenerated set of training data; and
logging the change that occurred to the given feature or the new feature.
8 . The system of claim 7 , wherein the constructing of the set of training data comprises:
combining the set of features with a browsing event data to generate combined data;
sampling the combined data to generate sampled data, wherein the sampled data is presented in a format having a structure that is agnostic to a feature source from which the set of features was fetched; and
joining the sampled data with old features by a back-filling process to generate training data designed to train one or more machine learning models.
9 . The system of claim 8 , wherein the one or more machine learning models are configured to be used to make predictions for respective one or more product areas of an electronic platform.
10 . The system of claim 8 , wherein the sampling comprises stratified sampling that applies weights to the set of features within the combined data, wherein the weights vary based on one or more of recency or frequency of browsing activity within the browsing event data.
11 . The system of claim 7 , wherein the set of features include (i) batch features and (ii) streaming features that are updated more often than the batch features.
12 . The system of claim 11 , wherein the batch features are stored in a first set of columns, wherein the streaming features are stored in a second set of columns, wherein the second set of columns are updated more often than the first set of columns.
13 . One or more non-transitory computer program products storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
fetching, over a network, a set of features from a plurality of feature sources;
constructing, using the set of features, a set of training data having a structure that is agnostic to a source from which the features were received;
detecting that (i) a change has occurred to a given feature, among the set of features that were used to construct the set of training data, that has not been logged in the set of features or (ii) a new feature exists that has not been logged in the set of features;
regenerating, prior to logging of the change that occurred to the given feature or the new feature that exists, the set of training data by joining the set of training data with the detected (i) change that occurred to the given feature or (ii) the new feature that exists, wherein the joining is performed based on a key of the training data; and
training, prior to logging the detected change to the given feature or the new feature that exists, a given machine learning model using the regenerated set of training data; and
logging the change that occurred to the given feature or the new feature.
14 . The one or more non-transitory computer program products of claim 13 , wherein the constructing of the set of training data comprises:
combining the set of features with a browsing event data to generate combined data;
sampling the combined data to generate sampled data, wherein the sampled data is presented in a format having a structure that is agnostic to a feature source from which the set of features was fetched; and
joining the sampled data with old features by a back-filling process to generate training data designed to train one or more machine learning models.
15 . The one or more non-transitory computer program products of claim 14 , wherein the one or more machine learning models are configured to be used to make predictions for respective one or more product areas of an electronic platform.
16 . The one or more non-transitory computer program products of claim 14 , wherein the sampling comprises stratified sampling that applies weights to the set of features within the combined data, wherein the weights vary based on one or more of recency or frequency of browsing activity within the browsing event data.
17 . The one or more non-transitory computer program products of claim 13 , wherein the set of features include (i) batch features and (ii) streaming features that are updated more often than the batch features.
18 . The one or more non-transitory computer program products of claim 17 , wherein the batch features are stored in a first set of columns, wherein the streaming features are stored in a second set of columns, wherein the second set of columns are updated more often than the first set of columns.