Data integrity checks
Aspects of the present disclosure relate to performing agnostic data integrity checks on source data, and based on the data integrity checks, generating a human-readable report that may be useable to identify specific errors or anomalies within the source data. Example embodiments involve systems and methods for performing the data integrity checks and generating the human-readable reports. For example, the method may include operations to ingest data from a source database through a data pipeline and into a local database, access the data from the data pipeline, determine a data type of the data, determine subtypes of data elements which make up the data, determine a count of each subtype, and generate a human-readable report, to be displayed at a client device.
1. A method comprising:
receiving data that comprises one or more data properties that characterize the data;
applying, by one or more processors, one or more data checks to the data to identify the one or more data properties, each data check of the one or more data checks including a set of Boolean checks, wherein the set of Boolean checks are applied in an order;
accessing a set of threshold values that correspond with the data based on the one or more data properties that characterize the data;
detecting an erroneous value among the data based on the set of threshold values;
causing display of a notification at a client device responsive to the detecting the erroneous value, the notification including an indication of the erroneous value.
2. The method of claim 1 , wherein the causing display of the notification further comprises:
generating a report that includes a display of at least a portion of the data that includes the erroneous value.
3. The method of claim 2 , wherein the erroneous value is visually distinguished from among the display of the portion of the data based on a visual property of the indication of the erroneous value.
4. The method of claim 2 , wherein the generating the report further comprises:
receiving an input that selects a set of report criteria; and
generating the report based on the set of report criteria.
5. The method of claim 1 , wherein the one or more data properties of the data include at least one selected from a group consisting of:
a source database of the data;
a unit of measurement associated with the data; and
a data type of the data.
6. The method of claim 1 , wherein the accessing the set of threshold values based on the one or more data properties of the data include:
accessing historical data based on the one or more data properties of the data; and
generating the set of threshold values based on the historical data.
7. The method of claim 1 , wherein the accessing the set of expected values based on the one or more data properties of the data includes:
receiving an input that assigns a set of expected values to the one or more data properties of the data; and
accessing the set of expected values based on the one or more data properties.
8. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
receiving data that comprises one or more data properties that characterize the data;
applying one or more data checks to the data to identify the one or more data properties, each data check of the one or more data checks including a set of Boolean checks, wherein the set of Boolean checks are applied in an order;
accessing a set of threshold values that correspond with the data based on the one or more data properties that characterize the data;
detecting an erroneous value among the data based on the set of threshold values;
causing display of a notification at a client device responsive to the detecting the erroneous value, the notification including an indication of the erroneous value.
9. The non-transitory machine-readable storage medium of claim 8 , wherein the causing display of the notification further comprises:
generating a report that includes a display of at least a portion of the data that includes the erroneous value.
10. The non-transitory machine-readable storage medium of claim 9 , wherein the erroneous value is visually distinguished from among the display of the portion of the data based on a visual property of the indication of the erroneous value.
11. The non-transitory machine-readable storage medium of claim 9 , wherein the generating the report further comprises:
receiving an input that selects a set of report criteria; and
generating the report based on the set of report criteria.
12. The non-transitory machine-readable storage medium of claim 8 , wherein the one or more data properties of the data include at least one selected from a group consisting of:
a source database of the data;
a unit of measurement associated with the data; and
a data type of the data.
13. The non-transitory machine-readable storage medium of claim 8 , wherein the accessing the set of threshold values based on the one or more data properties of the data include:
accessing historical data based on the one or more data properties of the data; and
generating the set of threshold values based on the historical data.
14. The non-transitory machine-readable storage medium of claim 8 , wherein the accessing the set of expected values based on the one or more data properties of the data includes:
receiving an input that assigns a set of expected values to the one or more data properties of the data; and
accessing the set of expected values based on the one or more data properties.
15. A system comprising:
processors; and
a memory storing instructions that, when executed by at least one processor among the processors, cause the system to perform operations comprising:
receiving data that comprises one or more data properties that characterize the data;
applying, by one or more processors, one or more data checks to the data to identify the one or more data properties, each data check of the one or more data checks including a set of Boolean checks, wherein the set of Boolean checks are applied in an order;
accessing a set of threshold values that correspond with the data based on the one or more data properties that characterize the data;
detecting an erroneous value among the data based on the set of threshold values;
causing display of a notification at a client device responsive to the detecting the erroneous value, the notification including an indication of the erroneous value.
16. The system of claim 15 , wherein the causing display of the notification further comprises:
generating a report that includes a display of at least a portion of the data that includes the erroneous value.
17. The system of claim 16 , wherein the erroneous value is visually distinguished from among the display of the portion of the data based on a visual property of the indication of the erroneous value.
18. The system of claim 16 , wherein the generating the report further comprises:
receiving an input that selects a set of report criteria; and
generating the report based on the set of report criteria.
19. The system of claim 15 , wherein the one or more data properties of the data include at least one selected from a group consisting of:
a source database of the data;
a unit of measurement associated with the data; and
a data type of the data.
20. The system of claim 15 , wherein the accessing the set of threshold values based on the one or more data properties of the data include:
accessing historical data based on the one or more data properties of the data; and
generating the set of threshold values based on the historical data.