Systems and methods for implementing data transformations in multiple execution contexts
A data transformation system for implementing reproducible and consistent data transformations in multiple execution contexts (batch, streaming, etc.) where the transformation function/logic initially acts on historical raw data to produce derived data to train a machine learning model. When the model is trained and deployed to handle streaming event data, the same transformation is reused to transform streaming data into the appropriate derived data for the model scoring, and later for a refit of the model.
1 . A system, comprising:
one or more processors in communication with:
a data gathering module;
a compute engine comprising at least one processor configured to execute transformation logic, wherein the transformation logic comprises a library of transformation logic units that are decoupled from functions involving retrieving data from one or more data sources and the compute engine is configured to be used in a plurality of execution contexts, the plurality of execution contexts comprising at least batch execution, application programming interface (API) processes and streaming execution;
a data catalog comprising memory configured to store logic metadata;
a derived data library; and
memory storing instructions that, when executed by the one or more processors, are configured to cause the system to:
receive, by the data gathering module, input data from a data source;
transform, according to the transformation logic and by the compute engine, the input data to produce a derived data output;
store, in the derived data library, the derived data output;
co-locate, as text in transformation function code, human- and machine-readable programmable logic metadata corresponding to execution of the transformation logic, wherein the logic metadata comprises an overridable pre-filter as a parameter for an input dataset to qualify the derived data output; and
one or more of train, score, and re-fit a machine learning model using the derived data output.
2 . The system of claim 1 , wherein batch historical data is utilized to train or re-fit the machine learning model.
3 . The system of claim 1 , wherein streaming data is utilized to score the machine learning model.
4 . The system of claim 3 , wherein the streaming data is real-time streaming data.
5 . The system of claim 1 , wherein the data gathering module comprises one or more of a batch data gatherer module, an application programming interface (API) data gatherer module, and a streaming data gatherer module.
6 . The system of claim 1 , wherein the compute engine is configured to implement a selectable unit of decoupled transformation logic, and wherein the input data is transformed to produce the derived data output according to a selected unit of decoupled transformation logic, wherein the decoupled transformation logic is a pure function and is independent of the data gathering module.
7 . The system of claim 1 , further comprising an orchestrator comprising memory in communication with the one or more processors, wherein the orchestrator is configured to detect a type of the input data from the data source and automatically select and trigger a corresponding type of reusable transformation logic, wherein the type comprises one of batch, API, or streaming.
8 . The system of claim 1 , wherein the data gathering module comprising memory is configured to receive raw input data from an authoritative data source.
9 . The system of claim 1 , wherein the logic metadata describes one or more of:
one or more derived data keys; and
a derived data output type.
10 . A method, comprising:
receiving, by a data gathering module, input data from a data source;
transforming, according to a unit of transformation logic and by a compute engine, the input data to produce a derived data output wherein the unit of transformation logic is decoupled from functions involving retrieving data from one or more data sources and the transformation logic is configured to be used in a plurality of execution contexts, the plurality of execution contexts comprising at least batch execution, application programming interface (API) processes and streaming execution;
storing, in a derived data library, the derived data output;
storing, in a data catalog, logic metadata corresponding to execution of the transformation logic; and
one or more of training, scoring, and re-fitting a machine learning model using the derived data output.
11 . The method of claim 10 , wherein batch historical data is utilized to train or re-fit the machine learning model, and wherein streaming data is utilized to score the machine learning model.
12 . The method of claim 11 , wherein the streaming data is real-time streaming data.
13 . The method of claim 10 , wherein the data gathering module comprises one or more of a batch data gatherer module, an application programming interface (API) data gatherer module, and a streaming data gatherer module.
14 . The method of claim 10 , wherein the logic metadata is configured to enable a nesting of one or more features to reduce duplication.
15 . A non-transitory computer-readable storage medium storing instructions that are configured to cause one or more processors to perform a method of:
receiving, by a data-gathering module, input data from a data source;
transforming, according to a unit of transformation logic and by a compute engine, the input data to produce a derived data output wherein the unit of transformation logic is configured to not retrieve the input data from the data source but is configured to reference the input data received by the data-gathering module and the compute engine is configured to be used in a plurality of execution contexts, the plurality of execution contexts comprising at least batch execution, application programming interface (API) processes and streaming execution;
storing, in a derived data library, the derived data output;
co-locating and storing as text in transformation function code, human- and machine-readable programmable logic metadata corresponding to the transformation logic, wherein the logic metadata comprises an overridable pre-filter as a parameter for an input dataset to qualify the derived data output; and
one or more of training, scoring, and re-fitting a machine learning model using the derived data output, wherein batch historical data is utilized to train or re-fit the machine learning model, and wherein streaming data is utilized to score the machine learning model.
16 . The system of claim 1 , wherein the one or more data sources comprise one or more of an authoritative data source and a database.
17 . The system of claim 1 , wherein the metadata provides contextual information regarding one or more of dependencies, lineage, quality and discovery.
18 . The method of claim 14 , wherein one of the one or more features comprises instructions on how to transform input data into new output data.
19 . The system of claim 1 , wherein operation of the compute engine is not affected by changes in data sources, storage mediums, models, or data retrieval technology used by the data gathering module to retrieve data.
20 . The system of claim 1 , wherein the transformation logic is modularized and allows mixing of execution contexts for chained processing.