Data lineage in a data pipeline
Various embodiments comprise systems and methods to monitor operations of a data pipeline. In some examples, a data pipeline receives data inputs, processes the data inputs, and responsively generates and transfers data outputs. Data monitoring circuitry monitors the operations of the data pipeline circuitry, identifies an input change between an initial one of the data inputs and a subsequent one of the data inputs, and identifies an output change between an initial one of the data outputs and a subsequent one of the data outputs. The data monitoring circuitry correlates the input change to the output change, determines a quality threshold for the output change based on the correlation, and determines when the output change falls below the quality threshold. When the output change falls below the quality threshold, the data monitoring circuitry generates and transfers an alert that indicates the input change and the output change.
1. A data pipeline monitoring system configured to monitor data quality in a data pipeline, the data pipeline monitoring system comprising:
data monitoring circuitry configured to monitor operations of the data pipeline, wherein the data pipeline receives data inputs, processes the data inputs, responsively generates data outputs, and transfers the data outputs;
the data monitoring circuitry configured to identify an input change between an initial one of the data inputs and a subsequent one of the data inputs, and identify an output change between an initial one of the data outputs and a subsequent one of the data outputs;
the data monitoring circuitry configured to correlate the input change to the output change, determine a quality threshold for the output change based on the correlation, and determine when the output change falls below the quality threshold; and
when the output change falls below the quality threshold, the data monitoring circuitry configured to generate and transfer an alert that indicates the input change and the output change.
2. The data pipeline monitoring system of claim 1 wherein:
the output change comprises a data volume change between the initial one of the data outputs and the subsequent one of the data outputs; and wherein:
the data monitoring circuitry is configured to correlate the input change to the output change comprises the data monitoring circuitry configured to correlate the input change to the data volume change.
3. The data pipeline monitoring system of claim 1 wherein:
the output change comprises a data schema change between the initial one of the data outputs and the subsequent one of the data outputs; and wherein:
the data monitoring circuitry is configured to correlate the input change to the output change comprises the data monitoring circuitry configured to correlate the input change to the data schema change.
4. The data pipeline monitoring system of claim 1 wherein:
the output change comprises a data value change between the initial one of the data outputs and the subsequent one of the data outputs; and wherein:
the data monitoring circuitry is configured to correlate the input change to the output change comprises the data monitoring circuitry configured to correlate the input change to the data value change.
5. The data pipeline monitoring system of claim 1 wherein:
the output change comprises a data type change between the initial one of the data outputs and the subsequent one of the data outputs; and wherein:
the data monitoring circuitry is configured to correlate the input change to the output change comprises the data monitoring circuitry configured to correlate the input change to the data type change.
6. The data pipeline monitoring system of claim 1 wherein the data monitoring circuitry is configured to identify the output change comprises the data monitoring circuitry configured to generate a histogram for the initial one of the data outputs, generate another histogram for the subsequent one of the data outputs, and identify the output change based on a difference between the histogram and the other histogram.
7. The data pipeline monitoring system of claim 6 wherein:
the histogram depicts data density for the initial one of the data outputs;
the other histogram represents data density for the subsequent one of the data outputs;
the data monitory circuitry is configured to identify the output change comprises the data monitory circuitry configured to identify the output change based on a difference between the data density for the initial one of the data outputs and the data density for the subsequent one of the data outputs.
8. A method of operating a data pipeline monitoring system to monitor data quality in a data pipeline, the data processing system comprising:
data monitoring circuitry monitoring operations of the data pipeline, wherein the data pipeline receives data inputs, processes the data inputs, responsively generates data outputs, and transferring the data outputs;
the data monitoring circuitry identifying an input change between an initial one of the data inputs and a subsequent one of the data inputs, and identifying an output change between an initial one of the data outputs and a subsequent one of the data outputs;
the data monitoring circuitry correlating the input change to the output change, determining a quality threshold for the output change based on the correlation, and determining when the output change falls below the quality threshold; and
when the output change falls below the quality threshold, the data monitoring circuitry generating and transferring an alert that indicates the input change and the output change.
9. The method of claim 8 wherein:
the output change comprises a data volume change between the initial one of the data outputs and the subsequent one of the data outputs; and wherein:
the data monitoring circuitry correlating the input change to the output change comprises the data monitoring circuitry correlating the input change to the data volume change.
10. The method of claim 8 wherein:
the output change comprises a data schema change between the initial one of the data outputs and the subsequent one of the data outputs; and wherein:
the data monitoring circuitry correlating the input change to the output change comprises the data monitoring circuitry correlating the input change to the data schema change.
11. The method of claim 8 wherein:
the output change comprises a data value change between the initial one of the data outputs and the subsequent one of the data outputs; and wherein:
the data monitoring circuitry correlating the input change to the output change comprises the data monitoring circuitry correlating the input change to the data value change.
12. The method of claim 8 wherein:
the output change comprises a data type change between the initial one of the data outputs and the subsequent one of the data outputs; and wherein:
the data monitoring circuitry correlating the input change to the output change comprises the data monitoring circuitry correlating the input change to the data type change.
13. The method of claim 8 wherein the data monitoring circuitry identifying the output change comprises the data monitoring circuitry generating a histogram for the initial one of the data outputs, generating another histogram for the subsequent one of the data outputs, and identifying the output change based on a difference between the histogram and the other histogram.
14. The method of claim 13 wherein:
the histogram represents data density for the initial one of the data outputs;
the other histogram represents data density for the subsequent one of the data outputs;
the data monitory circuitry identifying the output change comprises the data monitory circuitry identifying the output change based on a difference between the data density for the initial one of the data outputs and the data density for the subsequent one of the data outputs.
15. A non-transitory computer-readable medium storing instructions to monitor operations of a data pipeline, wherein the instructions, in response to execution by one or more processors, cause the one or more processors to drive a system to perform actions comprising:
monitoring the operations of the data pipeline wherein the data pipeline receives data inputs, processes the data inputs, and responsively generates and transfers data outputs;
identifying an input change between an initial one of the data inputs and a subsequent one of the data inputs;
identifying an output change between an initial one of the data outputs and a subsequent one of the data outputs;
correlating the input change to the output change;
determining a quality threshold for the output change based on the correlation;
determining when the output change falls below the quality threshold;
generating an alert that indicates the input change and the output change when the output change falls below the quality threshold; and
transferring the alert.
16. The non-transitory computer-readable medium of claim 15 wherein:
the output change comprises a data volume change between the initial one of the data outputs and the subsequent one of the data outputs; and wherein:
correlating the input change to the output change comprises correlating the input change to the data volume change.
17. The non-transitory computer-readable medium of claim 15 wherein:
the output change comprises a data schema change between the initial one of the data outputs and the subsequent one of the data outputs; and wherein:
correlating the input change to the output change comprises correlating the input change to the data schema change.
18. The non-transitory computer-readable medium of claim 15 wherein:
the output change comprises a data value change between the initial one of the data outputs and the subsequent one of the data outputs; and wherein:
correlating the input change to the output change comprises correlating the input change to the data value change.
19. The non-transitory computer-readable medium of claim 15 wherein:
the output change comprises a data type change between the initial one of the data outputs and the subsequent one of the data outputs; and wherein:
correlating the input change to the output change comprises correlating the input change to the data type change.
20. The non-transitory computer-readable medium of claim 15 , the operations further comprising:
generating a histogram for the initial one of the data outputs that represents data density for the initial one of the data outputs;
generating another histogram for the subsequent one of the data outputs that represents data density for the subsequent one of the data outputs; and wherein:
identifying the output change comprises identifying the output change based on a difference between the data density for the initial one of the data outputs and the data density for the subsequent one of the data outputs.