Data integrity checks
Aspects of the present disclosure relate to performing agnostic data integrity checks on source data, and based on the data integrity checks, generating a human-readable report that may be useable to identify specific errors or anomalies within the source data. Example embodiments involve systems and methods for performing the data integrity checks and generating the human-readable reports. For example, the method may include operations to ingest data from a source database through a data pipeline and into a local database, access the data from the data pipeline, determine a data type of the data, determine subtypes of data elements which make up the data, determine a count of each subtype, and generate a human-readable report, to be displayed at a client device.
1. A method comprising:
ingesting data through a data pipeline, the data pipeline located between a source database and a local database, the data comprising a plurality of data elements;
selecting a set of checks to apply to the data based on the source database of the data, the set of checks associated with the source database;
identifying a first data type associated with the one or more data elements of the plurality of data elements, based on the set of checks;
identifying the subtype of the first data type based on the set of checks;
calculating a count of the data elements of the identified subtype within the ingested data;
retrieving threshold values based on the first data type, the threshold values including a set of expected subtypes corresponding to the first data type, and a set of expected counts corresponding to the set of expected subtypes;
detecting a difference between the threshold values and the count of the data elements of the identified subtype, or between the set of expected subtypes and the identified subtype, within the ingested data; and
causing display of a notification at a client device in response to the detecting the difference, the notification including an indication of the difference between the expected counts and the count of the data elements, or the expected subtypes and the identified subtype.
2. The method of claim 1 , wherein the receiving the threshold value includes:
causing display of a graphical user interface at the client device, wherein the graphical user interface includes a set of input fields that correspond to a set of data elements, the plurality of data elements being among the set of data elements; and
receiving a user input defining the threshold value via an input field among the set of input fields.
3. The method of claim 1 , further comprising:
receiving a selection of the notification from the client device;
generating a human-readable report, the human-readable report including the subtype of each data element and the count of each subtype; and
causing display of the human-readable report at the client device in response to receiving the selection, the human-readable report including a presentation of the count of the data elements of the identified subtype within the ingested data, the threshold value, and the difference between the threshold value and the count of the data elements.
4. The method of claim 1 , wherein the set of checks include one or more Boolean checks.
5. The method of claim 1 , wherein the threshold value includes one or more of a numerical value and a percentage.
6. The method of claim 1 , wherein the method further comprises:
causing the difference to be visually distinguishing from the data.
7. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
ingesting data through a data pipeline, the data pipeline located between a source database and a local database, the data comprising a plurality of data elements;
selecting a set of checks to apply to the data based on the source database of the data, the set of checks associated with the source database;
identifying a first data type associated with the one or more data elements of the plurality of data elements, based on the set of checks;
identifying the subtype of the first data type based on the set of checks;
calculating a count of the data elements of the identified subtype within the ingested data;
retrieving threshold values based on the first data type, the threshold values including a set of expected subtypes corresponding to the first data type, and a set of expected counts corresponding to the set of expected subtypes;
detecting a difference between the threshold values and the count of the data elements of the identified subtype, or between the set of expected subtypes and the identified subtype, within the ingested data; and
causing display of a notification at a client device in response to the detecting the difference, the notification including an indication of the difference between the expected counts and the count of the data elements, or the expected subtypes and the identified subtype.
8. The non-transitory machine-readable storage medium of claim 7 , wherein the operations to receive the threshold value further comprises:
causing display of a graphical user interface at the client device, wherein the graphical user interface includes a set of input fields that correspond to a set of data elements, the plurality of data elements being among the set of data elements; and
receiving a user input defining the threshold value via an input field among the set of input fields.
9. The non-transitory machine-readable storage medium of claim 7 , wherein the operations further comprise:
receiving a selection of the notification from the client device;
generating a human-readable report, the human-readable report including the subtype of each data element and the count of each subtype; and
causing display of the human-readable report at the client device in response to receiving the selection, the human-readable report including a presentation of the count of the data elements of the identified subtype within the ingested data, the threshold value, and the difference between the threshold value and the count of the data elements.
10. The non-transitory machine-readable storage medium of claim 7 , wherein the set of checks include Boolean checks.
11. The non-transitory machine-readable storage medium of claim 7 , wherein the threshold value includes one or more of a numerical value and a percentage.
12. The non-transitory machine-readable storage medium of claim 7 , wherein the operations further comprise:
causing the difference to be visually distinguishing from the data.
13. A system comprising:
processors; and
a memory storing instructions that, when executed by at least one processor among the processors, cause the system to perform operations comprising:
ingesting data through a data pipeline, the data pipeline located between a source database and a local database, the data comprising a plurality of data elements;
selecting a set of checks to apply to the data based on the source database of the data, the set of checks associated with the source database;
identifying a first data type associated with the one or more data elements of the plurality of data elements, based on the set of checks;
identifying the subtype of the first data type based on the set of checks;
calculating a count of the data elements of the identified subtype within the ingested data;
retrieving threshold values based on the first data type, the threshold values including a set of expected subtypes corresponding to the first data type, and a set of expected counts corresponding to the set of expected subtypes;
detecting a difference between the threshold values and the count of the data elements of the identified subtype, or between the set of expected subtypes and the identified subtype, within the ingested data; and
causing display of a notification at a client device in response to the detecting the difference, the notification including an indication of the difference between the expected counts and the count of the data elements, or the expected subtypes and the identified subtype.
14. The system of claim 13 , wherein the instructions for the receiving the threshold value cause the system to perform operations comprising:
causing display of a graphical user interface at the client device, wherein the graphical user interface includes a set of input fields that correspond to a set of data elements, the plurality of data elements being among the set of data elements; and
receiving a user input defining the threshold value via an input field among the set of input fields.
15. The system of claim 14 , further comprising instructions that cause the system to perform operations further comprising:
receiving a selection of the notification from the client device;
generating a human-readable report, the human-readable report including the subtype of each data element and the count of each subtype; and
causing display of the human-readable report at the client device in response to receiving the selection, the human-readable report including a presentation of the count of the data elements of the identified subtype within the ingested data, the threshold value, and the difference between the threshold value and the count of the data elements.
16. The system of claim 14 , wherein the set of checks include Boolean checks.
17. The system of claim 13 , wherein the instructions cause the system to perform operations further comprising:
causing the difference to be visually distinguishing from the data.