Machine learning framework and method for using the same
A machine learning framework and method for using the same are described. In one embodiment, the method for processing data with a machine learning framework comprises creating a plurality of features as independent features, each feature of the plurality features being based on one or more events that model a plurality of records related to payment processing information, creating a final feature that groups the plurality of features together, such that each feature of the plurality of features represents a sub-feature of the final feature, compiling the plurality of features and the final feature, computing, using a computing platform, each of the plurality of features as a separate job, including sending network related communications to access the payment processing information from one or more remote storage locations, and computing, using a computing platform, the final feature separately from computing the plurality of features, including grouping results of running each of the plurality of features together.
1 . A computer-implemented method for processing data, the method comprising:
defining, at a computing platform, a plurality of features as independent features, wherein each feature of the plurality of features is defined by a key and a value type, and each feature of the plurality of features is further defined to execute a mapper that converts data records generated by disparate systems and having the key and value type to a stream of events of a same event object type having a unified data format;
executing, by the computing platform, the mapper defined for each feature of the plurality of features comprising: accessing a plurality of data records generated by the disparate systems and generating the stream of events of the same event object type in the unified data format from the plurality of data records to model a temporal stream of time-stamped records, the mapper adding a read-only timestamp to each of the stream of events;
defining, at the computing platform, a final feature based on the plurality of features, such that each feature of the plurality of features is used for computation of the final feature;
compiling, by the computing platform, the plurality of features and the final feature into a plurality of jobs, wherein each job of the plurality of jobs is compiled by generating a combiner that aggregates events from a corresponding stream of events; and
executing each of the plurality of jobs using the computing platform, wherein executing a job comprises:
executing, using the computing platform, combiners compiled for each of the plurality of features to generate a value for each of the plurality of features, including sending network related communications to access one or more of the time-stamped records, based on a key that corresponds to said each of the plurality of features, from one or more remote storage locations, and
executing, using the computing platform, a combiner compiled for the final feature to generate a value for the final feature separately from computing the values for the plurality of features, including grouping together the values for the plurality of features.
2 . The method defined in claim 1 wherein each of the plurality of features is run as a separate job comprising performing jobs for two or more features of the plurality of features in parallel in real time or as a batch computing process.
3 . The method defined in claim 1 wherein the features of the plurality of features and final features are user specified via an application programming interface (API).
4 . The method defined in claim 1 further comprising updating one or more predictive models using a result from computing the final feature.
5 . The method defined in claim 1 further comprising:
after computing the value for the final feature, storing separately, for each feature in the plurality of features, results from computing a corresponding value computed for said each feature; and
enabling access to one or more of the results from computing each feature of the plurality of features for individual use by another feature other than the final feature.
6 . The method defined in claim 1 wherein data from computing the plurality of features is stored in columnar format, and wherein grouping results of running each of the plurality of features together as part of computing the final feature is made, at least in part, based on the columnar format.
7 . The method defined in claim 1 wherein at least one event of the stream of events of at least one feature of the plurality of features performs a lookup to attach a value of another feature to a feed in the at least one event.
8 . The method defined in claim 7 wherein the lookup retains all information associated with an event.
9 . The method defined in claim 7 wherein the lookup retains only a key associated with an event.
10 . The method defined in claim 1 further comprising obtaining at least one feature of the plurality of features by aggregating events from a corresponding stream of events.
11 . A system comprising:
a network interface;
a memory to store instructions;
one or more processors coupled to the memory and the network interface to execute the stored instructions to:
define a plurality of features as independent features, wherein each feature of the plurality features is defined by a key and a value type, and each feature of the plurality of features is further defined to execute a mapper that converts data records generated by disparate systems and having the key and value type to a stream of events of a same event object type having a unified data format;
execute the mapper defined for each feature of the plurality of features to access a plurality of data records generated by the disparate systems and generate the stream of events of the same event object type in the unified data format from the plurality of data records to model a temporal stream of time-stamped records, the mapper adding a read-only timestamp to each of the stream of events;
define a final feature that groups the plurality of features together, such that each feature of the plurality of features represents a sub-feature of the final feature;
compile the plurality of features and the final feature into a plurality of jobs, wherein each job of the plurality of jobs is compiled by generating a combiner that aggregates events from a corresponding stream of events; and
execute each of the plurality of jobs using a computing platform, wherein execution of a job comprises the one or more processors to:
execute combiners compiled for jobs associated with the plurality of features to generate a value for each of the plurality of features, including sending network related communications to access one or more of the time-stamped records, based on a key that corresponds to said each of the plurality of features, from one or more remote storage locations, and
execute a combiner compiled for the final feature to generate a value for the final feature separately from computing the values for the plurality of features, including grouping together the values for the plurality of features.
12 . The system defined in claim 11 wherein each of the plurality of features is computed as a separate job by performing jobs for two or more features of the plurality of features in parallel in real time or as a batch computing process.
13 . The system defined in claim 11 wherein the features of the plurality of features and final features are user specified via an application programming interface (API).
14 . The system defined in claim 11 wherein the one or more processors are operable to update one or more predictive models using a result from computing the final feature.
15 . The system defined in claim 11 wherein the one or more processors are operable to, after computing the value for the final feature,
store separately, for each feature in the plurality of features, results from computing a corresponding value computed for said each feature; and
enable access to one or more of the results from computing each feature of the plurality features for individual use by another feature other than the final feature.
16 . The system defined in claim 11 wherein data from computing the plurality of features is stored in columnar format, and wherein grouping results of running each of the plurality of features together as part of computing the final feature is made, at least in part, based on the columnar format.
17 . The system defined in claim 11 wherein at least one event of the stream of events of at least one feature of the plurality of features performs a lookup to attach a value of another feature to a feed in the at least one event.
18 . The system defined in claim 17 wherein the lookup retains all information associated with an event.
19 . The system defined in claim 17 wherein the lookup retains only a key associated with an event.
20 . The system defined in claim 11 wherein the one or more processors are operable to obtain at least one feature of the plurality of features by aggregating events from a corresponding stream of events.
21 . One or more non-transitory computer readable storage media having instructions stored thereupon which, when executed by a computing platform having at least a processor and a memory therein, cause the computing platform to perform operations comprising:
defining, at the computing platform, a plurality of features as independent features, wherein each feature of the plurality of features is defined by a key and a value type, and each feature of the plurality of features is further defined to execute a mapper that converts data records generated by disparate systems and having the key and value type to a stream of events of a same event object type having a unified data format;
executing, by the computing platform, the mapper defined for each feature of the plurality of features comprising: accessing, by the mapper, a plurality of data records generated by the disparate systems and generating, by the mapper, the stream of events of the same event object type in the unified data format from the plurality of data records to model a temporal stream of time-stamped records, the mapper adding a read-only timestamp to each of the stream of events;
defining, at the computing platform, a final feature based on the plurality of features, such that each feature of the plurality of features is used for computation of the final feature;
compiling, by the computing platform, the plurality of features and the final feature into a plurality of jobs, wherein each job of the plurality of jobs is compiled by generating a combiner that aggregates events from a corresponding stream of events; and
executing each of the plurality of jobs using the computing platform, wherein executing a job comprises:
executing, using the computing platform, combiners compiled for each of the plurality of features to generate a value for each of the plurality of features, including sending network related communications to access one or more of the time-stamped records, based on a key that corresponds to said each of the plurality of features, from one or more remote storage locations, and
executing, using the computing platform, a combiner compiled for the final feature to generate a value for the final feature separately from computing the values for the plurality of features, including grouping together the values for the plurality of features.
22 . The storage media defined in claim 21 wherein each of the plurality of features is run as a separate job comprising performing jobs for two or more features of the plurality of features in parallel in real time or as a batch computing process.
23 . The storage media defined in claim 21 wherein the features of the plurality of features and final features are user specified via an application programming interface (API).
24 . The storage media defined in claim 21 further comprising updating one or more predictive models using a result from computing the final feature.
25 . The storage media defined in claim 21 wherein at least one event of the stream of events of at least one feature of the plurality of features performs a lookup to attach a value of another feature to a feed in the at least one event.