Consistency monitoring of data in a data pipeline
Various embodiments comprise systems and methods to maintain data consistency in a data pipeline. In some examples, a computing system comprises data monitoring circuitry that monitors the operations of the data pipeline. The data pipeline receives input data, processes the input data, and generates output data. The data monitoring circuitry receives and processes the output data sets to identify changes between the output data sets. The data monitoring circuitry generates a consistency score based on the changes that indicates a similarity level between the output data sets. The data monitoring circuitry determines when the consistency score exceeds a threshold value. When the consistency score exceeds the threshold value, the data monitoring circuitry generates and transfers an alert that indicates ones of the output data sets that exceeded the threshold value.
1. A data pipeline monitoring system configured to maintain data consistency in a data pipeline, the data pipeline monitoring system comprising:
data monitoring circuitry configured to monitor operations of the data pipeline wherein the data pipeline receives input data sets, processes the input data sets, and responsively generates and transfers output data sets;
the data monitoring circuitry configured to receive the output data sets, process the output data sets to identify changes between the output data sets, utilize a Sigmoid function to calculate weighted sums of data values, data records, and data schemas for the output data sets, generate a consistency score for the output data sets based on the changes and the weighted sums, and determine when the consistency score exceeds a threshold value, wherein the consistency score indicates a similarity level between the output data sets; and
when the consistency score exceeds the threshold value, the data monitoring circuitry configured to generate and transfer an alert that indicates ones of the output data sets that exceeded the threshold value.
2. The data pipeline monitoring system of claim 1 , wherein the data monitoring circuitry is configured to identify changes between the output data sets and generate the consistency score comprises the data monitoring circuitry configured to determine null value rates for the output data sets, compare the null value rates between different ones of the output data sets, and generate the consistency score based on the comparison.
3. The data pipeline monitoring system of claim 1 , wherein the data monitoring circuitry is configured to identify changes between the output data sets and generate the consistency score comprises the data monitoring circuitry configured to determine data entropies for the output data sets, compare the data entropies between different ones of the output data sets, and generate the consistency score based on the comparison.
4. The data pipeline monitoring system of claim 1 , wherein the data monitoring circuitry is configured to identify changes between the output data sets and generate the consistency score comprises the data monitoring circuitry configured to determine average data values for the output data sets, compare the average data values between different ones of the output data sets, and generate the consistency score based on the comparison.
5. The data pipeline monitoring system of claim 1 , wherein the alert indicates a severity level for the identified changes.
6. The data pipeline monitoring system of claim 5 , wherein the alert indicates a date of operation the identified changes occurred.
7. The data pipeline monitoring system of claim 1 , wherein the threshold value comprises a user configurable threshold value.
8. A method of operating a data pipeline monitoring system configured to maintain data consistency in a data pipeline, the method comprising:
data monitoring circuitry monitoring operations of the data pipeline wherein the data pipeline receives input data sets, processes the input data sets, and responsively generates and transfers output data sets;
the data monitoring circuitry receiving the output data sets, processing the output data sets to identify changes between the output data sets, utilizing a Sigmoid function to calculate weighted sums of data values, data records, and data schemas for the output data sets, generating a consistency score for the output data sets based on the changes and the weighted sums, and determining when the consistency score exceeds a threshold value, wherein the consistency score indicates a similarity level between the output data sets; and
when the consistency score exceeds the threshold value, the data monitoring circuitry generating and transferring an alert that indicates ones of the output data sets that exceeded the threshold value.
9. The method of claim 8 , wherein the data monitoring circuitry identifying changes between the output data sets and generating the consistency score comprises the data monitoring circuitry determining null value rates for the output data sets, comparing the null value rates between different ones of the output data sets, and generating the consistency score based on the comparison.
10. The method of claim 8 , wherein the data monitoring circuitry identifying changes between the output data sets and generating the consistency score comprises the data monitoring circuitry determining data entropies for the output data sets, comparing the data entropies between different ones of the output data sets, and generating the consistency score based on the comparison.
11. The method of claim 8 , wherein the data monitoring circuitry identifying changes between the output data sets and generating the consistency score comprises the data monitoring circuitry determining average data values for the output data sets, comparing the average data values between different ones of the output data sets, and generating the consistency score based on the comparison.
12. The method of claim 8 , wherein the alert indicates a severity level for the identified changes.
13. The method of claim 12 , wherein the alert indicates a date of operation the identified changes occurred.
14. The method of claim 8 , wherein the threshold value comprises a user configurable threshold value.
15. A non-transitory computer-readable medium storing instructions to maintain data consistency in a data pipeline, wherein the instructions, in response to execution by one or more processors, cause the one or more processors to drive a system to perform operations comprising:
monitoring operations of the data pipeline wherein the data pipeline receives input data sets, processes the input data sets, and responsively generates and transfers output data sets;
receiving the output data sets;
processing the output data sets to identify changes between the output data sets;
utilizing a Sigmoid function to calculate weighted sums of data values, data records, and data schemas for the output data sets;
generating a consistency score for the output data sets based on the changes and the weighted sums, wherein the consistency score indicates a similarity level between the output data sets;
determining when the consistency score exceeds a threshold value;
generating an alert that indicates ones of the output data sets that exceeded the threshold value when the consistency score exceeds the threshold value; and
transferring the alert.
16. The non-transitory computer-readable medium of claim 15 , the operations further comprising:
determining null value rates for the output data sets;
comparing the null value rates between different ones of the output data sets; and
generating the consistency score based on the comparison.
17. The non-transitory computer-readable medium of claim 15 , the operations further comprising:
determining data entropies rates for the output data sets;
comparing the data entropies between different ones of the output data sets; and
generating the consistency score based on the comparison.
18. The non-transitory computer-readable medium of claim 15 , the operations further comprising:
determining data values rates for the output data sets;
comparing the data values between different ones of the output data sets; and
generating the consistency score based on the comparison.
19. The non-transitory computer-readable medium of claim 15 , wherein the alert indicates a severity level for the identified changes and a date of operation the identified changes occurred.
20. The non-transitory computer-readable medium of claim 15 , wherein the threshold value comprises a user configurable threshold value.