Data drift handling

1 . A method comprising:

accessing, using one or more processors, a data pipeline, the data pipeline comprising a plurality of stages, wherein the data pipeline is executed on a cluster-computing framework comprising a group of inter-connected computers that work together to perform computationally intensive tasks;

receiving a fixed set of schema fields for the data pipeline to function without error;

for each stage of the plurality of stages in the data pipeline starting from one or more destination stages and traversing backward through the data pipeline until reaching an origin stage:

identifying schema fields for processing data in the data pipeline; and

generating a set of stage schema fields comprising the identified stage schema fields in the stage;

in response to detecting the origin stage, generating a set of pipeline schema fields, the set of pipeline schema fields comprising a union of the generated sets of stage schema fields, the union of the generated sets of stage schema fields representing pipeline schema fields required for the data pipeline to function without error,

storing the set of pipeline schema fields;

comparing the set of pipeline schema fields to the fixed set of schema fields;

comparing data in the data pipeline to the set of pipeline schema fields;

in response to detecting that the set of pipeline schema fields is compatible with the data in the data pipeline, writing the data to a first target destination, wherein the first target destination is configured to store data that meets the compatibility requirements of the fixed set of schema fields and is intended for further processing or use in downstream applications;

in response to detecting that the data in the data pipeline corresponding to at least one pipeline schema field in the set of pipeline schema fields does not include all of the fields of data specified by the at least one pipeline schema field, writing the data to a second target location, wherein the second target location is configured to store data that does not meet the compatibility requirements of the fixed set of schema fields and is intended for diagnostic or corrective actions to address schema incompatibilities; and

removing the data in the second target location from the data pipeline.

2 . The method of claim 1 , wherein the set of pipeline schema fields represents minimum required schema fields for the data pipeline.

3 . The method of claim 1 , further comprising:

updating schema fields for the data pipeline based on the set of pipeline schema fields.

4 . The method of claim 3 , wherein the set of pipeline schema fields are stored as offset information, the offset information providing information on how to restart the data pipeline from a last batch of data that was processed.

5 . The method of claim 1 , further comprising:

in response to detecting that the set of pipeline schema fields is compatible with the data in the data pipeline:

reading the data in the data pipeline; and

processing the data based the set of pipeline schema fields, wherein the processing occurs at the first target destination configured for downstream applications.

6 . The method of claim 1 , further comprising:

in response to detecting that at least one pipeline schema field in the set of pipeline schema fields is not detected in the data in the data pipeline, detecting a failure in the data pipeline; and

wherein the second target location is a database configured to store data for diagnostic or corrective actions to address schema incompatibilities.

7 . A system comprising:

one or more processors; and

a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

accessing a data pipeline, the data pipeline comprising a plurality of stages, wherein the data pipeline is executed on a cluster-computing framework comprising a group of inter-connected computers that work together to perform computationally intensive tasks;

receiving a fixed set of schema fields for the data pipeline to function without error;

for each stage of the plurality of stages in the data pipeline starting from one or more destination stages and traversing backward through the data pipeline until reaching an origin stage:

identifying stage schema fields for processing data in the data pipeline; and

generating a set of stage schema fields comprising the identified stage schema fields in the stage;

in response to detecting the origin stage, generating a set of pipeline schema fields, the set of pipeline schema fields comprising a union of the generated sets of stage schema fields, the union of the generated sets of stage schema fields representing pipeline schema fields required for the data pipeline to function without error;

storing the set of pipeline schema fields;

comparing the set of pipeline schema fields to the fixed set of schema fields; and

comparing data in the data pipeline to the set of pipeline schema fields;

in response to detecting that the set of pipeline schema fields is compatible with the data in the data pipeline, writing the data to a first target destination, wherein the first target destination is configured to store data that meets the compatibility requirements of the fixed set of schema fields and is intended for further processing or use in downstream applications;

in response to detecting that the data in the data pipeline corresponding to at least one pipeline schema field in the set of pipeline schema fields does not include all of the fields of data specified by the at least one pipeline schema field, writing the data to a second target location, wherein the second target location is configured to store data that does not meet the compatibility requirements of the fixed set of schema fields and is intended for diagnostic or corrective actions to address schema incompatibilities; and

removing the data in the second target location from the data pipeline.

8 . The system of claim 7 , wherein the set of pipeline schema fields represents a minimum required schema fields for the data pipeline.

9 . The system of claim 7 , wherein the operations further comprise:

updating schema fields for the data pipeline based on the set of pipeline schema fields.

10 . The system of claim 9 , wherein the set of pipeline schema fields are stored as offset information, the offset information providing information on how to restart the data pipeline from a last batch of data that was processed.

11 . The system of claim 7 , wherein the operations further comprise:

in response to detecting that the set of pipeline schema fields is compatible with the data in the data pipeline:

reading the data in the data pipeline; and

processing the data based the set of pipeline schema fields, wherein the processing occurs at the first target destination configured for downstream applications.

12 . The system of claim 7 , wherein the operations further comprise:

in response to detecting that at least one pipeline schema field in the set of pipeline schema fields is not detected in the data in the data pipeline, detecting a failure in the data pipeline; and

wherein the second target location is a database configured to store data for diagnostic or corrective actions to address schema incompatibilities.

13 . A non-transitory machine storage medium storing instructions that when executed by one or more processors of a machine, cause the machine to perform operations comprising:

accessing a data pipeline, the data pipeline comprising a plurality of stages, wherein the data pipeline is executed on a cluster-computing framework comprising a group of inter-connected computers that work together to perform computationally intensive tasks;

receiving a fixed set of schema fields for the data pipeline to function without error;

for each stage of the plurality of stages in the data pipeline starting from one or more destination stages and traversing backward through the data pipeline until reaching an origin stage:

identifying stage schema fields for processing data in the data pipeline; and

generating a set of stage schema fields comprising the identified stage schema fields in the stage;

in response to detecting the origin stage, generating a set of pipeline schema fields, the set of pipeline schema fields comprising a union of the generated sets of stage schema fields, the union of the generated sets of stage schema fields representing pipeline schema fields required for the data pipeline to function without error;

storing the set of pipeline schema fields;

comparing the set of pipeline schema fields to the fixed set of schema fields; and comparing data in the data pipeline to the set of pipeline schema fields;

in response to detecting that the set of pipeline schema fields is compatible with the data in the data pipeline, writing the data to a first target destination, wherein the first target destination is configured to store data that meets the compatibility requirements of the fixed set of schema fields and is intended for further processing or use in downstream applications;

in response to detecting that the data in the data pipeline corresponding to at least one pipeline schema field in the set of pipeline schema fields does not include all of the fields of data specified by the at least one pipeline schema field, writing the data to a second target location, wherein the second target location is configured to store data that does not meet the compatibility requirements of the fixed set of schema fields and is intended for diagnostic or corrective actions to address schema incompatibilities; and

removing the data in the second target location from the data pipeline.

14 . The non-transitory machine storage medium of claim 13 , wherein the set of pipeline schema fields represents minimum required schema fields for the data pipeline.

15 . The non-transitory machine storage medium of claim 13 , wherein the operations further comprise:

updating schema fields for the data pipeline based on the set of pipeline schema fields.

16 . The non-transitory machine storage medium of claim 15 , wherein the set of pipeline schema fields are stored as offset information, the offset information providing information on how to restart the data pipeline from a last batch of data that was processed.

17 . The non-transitory machine storage medium of claim 13 , wherein the operations further comprise:

in response to detecting that at least one pipeline schema field in the set of pipeline schema fields is not detected in the data in the data pipeline, detecting a failure in the data pipeline; and

wherein the second target location is a database configured to store data for diagnostic or corrective actions to address schema incompatibilities.