Data processing pipeline with data regression framework
An example embodiment provides a method that includes one or more of executing a sequence of job steps for a job by a sequence of job step executors within a data processing pipeline, recording current job execution data generated by the sequence of job step executors during the executing of the sequence of job steps, determining whether the executing of the sequence of job steps includes a deviation based on comparing the current job execution data to historical job execution data of the data processing pipeline stored in a data store, wherein the deviation is determined by comparing a current step execution dataset to a recorded step execution dataset in the data store, and generating a failure alert when the deviation is included.
1 . A system, comprising:
a data store configured to store historical job execution data of a data processing pipeline; and
a processor communicatively coupled to the data store, wherein the processor is configured to:
receive a request from a pipeline manager executing on a host platform to execute a job comprising a sequence of job steps,
in response to the request, instantiating a sequence of job step executors within the data processing pipeline before execution of the job, wherein the instantiating comprises resolving dependencies of the job step executors in the sequence of job step executors,
record current job execution data generated by the sequence of job step executors during the execution of the job,
perform a comparison of a current job step execution dataset to the historical job execution data in the data store,
identify a deviation in the sequence of job steps based on the comparison, and
in response to the deviation being identified, generate a failure alert and capture metadata about a job step with the deviation from the pipeline manager;
wherein the sequence of job steps is determined using a Directed Acyclic Graph (DAG) of executors generated based on the dependencies of the job step executors.
2 . The system of claim 1 , wherein the sequence of job steps is defined in one or more configuration files.
3 . The system of claim 1 , wherein the DAG comprises recording executors that record the current job execution data.
4 . The system of claim 1 , wherein the DAG comprises compare executors that compare the current job step execution dataset to the historical job execution data.
5 . The system of claim 1 , wherein the processor is further configured to:
record the deviation.
6 . The system of claim 1 , wherein the processor is further configured to:
generate a data lineage in response to the deviation being identified.
7 . A method, comprising:
receiving a request from a pipeline manager executing on a host platform to execute a job comprising a sequence of job steps;
in response to the request, instantiating a sequence of job step executors within the data processing pipeline before execution of the job, wherein the instantiating comprises resolving dependencies of the job step executors in the sequence of job step executors;
executing the sequence of job steps;
recording current job execution data generated by the sequence of job step executors when the job is executed;
comparing a current job step execution dataset to the historical job execution data in the data store;
identifying a deviation in the sequence of job steps based on the comparison; and
in response to the deviation being identified, generating a failure alert and capture metadata about a job step with the deviation from the pipeline manager;
wherein the sequence of job steps is determined using a Directed Acyclic Graph (DAG) of executors generated based on the dependencies of the job step executors.
8 . The method of claim 7 , wherein the sequence of job steps is defined in one or more configuration files.
9 . The method of claim 7 , wherein the DAG comprises recording executors that record the current job execution data.
10 . The method of claim 7 , wherein the DAG comprises compare executors that compare the current job step execution dataset to the historical job execution data.
11 . The method of claim 7 , further comprising:
recording the deviation.
12 . The method of claim 7 , further comprising:
generating a data lineage in response to the deviation being identified.
13 . A non-transitory computer storage medium comprising instructions that, when executed by a processor, cause the processor to perform:
receiving a request from a pipeline manager executing on a host platform to execute a job comprising a sequence of job steps;
in response to the request, instantiating a sequence of job step executors within the data processing pipeline before execution of the job, wherein the instantiating comprises resolving dependencies of the job step executors in the sequence of job step executors;
executing the sequence of job steps;
recording current job execution data generated by the sequence of job step executors when the job is executed;
comparing a current job step execution dataset to the historical job execution data in the data store;
identifying a deviation in the sequence of job steps based on the comparison; and
in response to the deviation being identified, generating a failure alert and capture metadata about a job step with the deviation from the pipeline manager;
wherein the sequence of job steps is determined using a Directed Acyclic Graph (DAG) of executors generated based on the dependencies of the job step executors.
14 . The non-transitory computer storage medium of claim 13 , wherein the sequence of job steps is defined in one or more configuration files.
15 . The non-transitory computer storage medium of claim 13 , wherein the DAG comprises recording executors that record the current job execution data.
16 . The non-transitory computer storage medium of claim 13 , wherein the DAG comprises compare executors that compare the current job step execution dataset to the historical job execution data.
17 . The non-transitory computer storage medium of claim 13 , wherein the instructions cause the processor to perform:
recording the deviation; and
generating a data lineage in response to the deviation being identified.