Data drift handling

1. A method for handling data drift in a data pipeline, the method comprising:

accessing, using one or more processors, the data pipeline, the data pipeline comprising a plurality of stages;

receiving user input indicating a fixed set of schema fields for the data pipeline to function without error;

for each stage of the plurality of stages in the data pipeline:

identifying stage schema fields for processing data in the data pipeline; and

generating a set of stage schema fields comprising the identified stage schema fields in the stage;

in response to detecting an origin stage, the origin stage indicating a data source of the data pipeline, generating a set of pipeline schema fields, the set of pipeline schema fields comprising a union of the generated sets of stage schema fields;

storing the set of pipeline schema fields;

comparing the set of pipeline schema fields to the fixed set of schema fields; and

comparing data in the data pipeline to the set of pipeline schema fields;

in response to detecting that the set of pipeline schema fields is compatible with the data in the data pipeline, writing the data to a first target destination; and

in response to detecting that at least one pipeline schema field in the set of pipeline schema fields is not detected in the data in the data pipeline, writing the data to a second target location for analysis.

2. The method of claim 1 , wherein the set of pipeline schema fields represents minimum required schema fields for the data pipeline.

3. The method of claim 1 , further comprising:

updating schema fields for the data pipeline based on the set of pipeline schema fields.

4. The method of claim 3 , wherein the set of pipeline schema fields are stored as offset information, the offset information providing information on how to restart the data pipeline from a last batch of data that was processed.

5. The method of claim 1 , wherein:

the fixed set of schema fields comprises a minimum number of schema fields for the data pipeline to function without error.

6. The method of claim 1 , further comprising:

in response to detecting that the set of pipeline schema fields is compatible with the data in the data pipeline;

reading the data in the data pipeline; and

processing the data based the set of pipeline schema fields.

7. The method of claim 1 , further comprising:

in response to detecting that at least one pipeline schema field in the set of pipeline schema fields is not detected in the data in the data pipeline, detecting a failure in the data pipeline; and

wherein the second target location is a database.

8. A computing apparatus, the computing apparatus comprising:

one or more processors; and

a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

accessing a data pipeline comprising a plurality of stages;

receiving user input indicating a fixed set of schema fields for the data pipeline to function without error;

for each stage of the plurality of stages in the data pipeline:

identifying stage schema fields for processing data in the data pipeline; and

generating a set of stage schema fields comprising the identified stage schema fields in the stage;

in response to detecting an origin stage, the origin stage indicating a data source of the data pipeline, generating a set of pipeline schema fields, the set of pipeline schema fields comprising a union of the generated sets of stage schema fields;

storing the set of pipeline schema fields;

comparing the set of pipeline schema fields to the fixed set of schema fields;

comparing data in the data pipeline to the set of pipeline schema fields;

in response to detecting that the set of pipeline schema fields is compatible with the data in the data pipeline, writing the data to a first target destination; and

in response to detecting that at least one pipeline schema field in the set of pipeline schema fields is not detected in the data in the data pipeline, writing the data to a second target location for analysis.

9. The computing apparatus of claim 8 , wherein the set of pipeline schema fields represents a minimum required schema fields for the data pipeline.

10. The computing apparatus of claim 8 , wherein the operations further comprise:

updating schema fields for the data pipeline based on the set of pipeline schema fields.

11. The computing apparatus of claim 10 , wherein the set of pipeline schema fields are stored as offset information, the offset information providing information on how to restart the data pipeline from a last batch of data that was processed.

12. The computing apparatus of claim 8 , wherein the fixed set of schema fields comprises a minimum number of schema fields for the data pipeline to function without error.

13. The computing apparatus of claim 8 , wherein the operations further comprise:

in response to detecting that the set of pipeline schema fields is compatible with the data in the data pipeline:

reading the data in the data pipeline; and

processing the data based the set of pipeline schema fields.

14. The computing apparatus of claim 8 , wherein the operations further comprise:

in response to detecting that at least one pipeline schema field in the set of pipeline schema fields is not detected in the data in the data pipeline, detecting a failure in the data pipeline; and

wherein the second target location is a database.

15. A non-transitory machine storage medium storing instructions that when executed by one or more processors of a machine, cause the machine to perform operations comprising:

accessing a data pipeline comprising a plurality of stages;

receiving user input indicating a fixed set of schema fields for the data pipeline to function without error;

for each stage of the plurality of stages in the data pipeline:

identifying stage schema fields for processing data in the data pipeline;

generating a set of stage schema fields comprising the identified stage schema fields in the stage;

in response to detecting an origin stage, the origin stage indicating a data source of the data pipeline, generating a set of pipeline schema fields, the set of pipeline schema fields comprising a union of the generated sets of stage schema fields;

storing the set of pipeline schema fields;

comparing the set of pipeline schema fields to the fixed set of schema fields;

comparing data in the data pipeline to the set of pipeline schema fields;

in response to detecting that the set of pipeline schema fields is compatible with the data in the data pipeline, writing the data to a first target destination; and

in response to detecting that at least one pipeline schema field in the set of pipeline schema fields is not detected in the data in the data pipeline, writing the data to a second target location for analysis.

16. The non-transitory machine storage medium of claim 15 , wherein the set of pipeline schema fields represents minimum required schema fields for the data pipeline.

17. The non-transitory machine storage medium of claim 15 , wherein the operations further comprise:

updating schema fields for the data pipeline based on the set of pipeline schema fields.

18. The non-transitory machine storage medium of claim 17 , wherein the set of pipeline schema fields are stored as offset information, the offset information providing information on how to restart the data pipeline from a last batch of data that was processed.

19. The non-transitory machine storage medium of claim 15 , wherein the fixed set of schema fields is a minimum number of required schema fields for the data pipeline to function without error.

20. The non-transitory machine storage medium of claim 15 , wherein the operations further comprise:

in response to detecting that at least one pipeline schema field in the set of pipeline schema fields is not detected in the data in the data pipeline, detecting a failure in the data pipeline; and

wherein the second target location is a database.