Data integrity checks
Aspects of the present disclosure relate to performing agnostic data integrity checks on source data, and based on the data integrity checks, generating a human-readable report that may be useable to identify specific errors or anomalies within the source data. Example embodiments involve systems and methods for performing the data integrity checks and generating the human-readable reports. For example, the method may include operations to ingest data from a source database through a data pipeline and into a local database, access the data from the data pipeline, determine a data type of the data, determine subtypes of data elements which make up the data, determine a count of each subtype, and generate a human-readable report, to be displayed at a client device.
1. A method comprising:
receiving, at a local database, data from a source database among a plurality of databases;
determining the source database of the data from among the plurality of databases based on an attribute of the data;
accessing historical data associated with the source database of the data, the historical data comprising a set of expected values that correspond with the source database of the data;
determining a threshold value based on the set of expected values of the historical data associated with the source database;
performing a comparison of the data from the source database with the threshold value;
detecting an erroneous value among the data from the source database based on the comparison of the data from the source database with the threshold value;
causing display of a notification that indicates the erroneous value at a client device in response to the detecting the erroneous value; and
generating a report that comprises a display of at least a portion of the data from the source database.
2. The method of claim 1 , wherein the display of the data from the source database includes a presentation of the erroneous value, wherein the presentation of the erroneous value is visually distinguished from the display of the data.
3. The method of claim 1 , wherein the data from the source database comprises a data type, and the accessing the set of expected values includes:
accessing the set of expected values based on the source database and the data type of the data.
4. The method of claim 1 , wherein the set of expected values include a set of threshold values, and the detecting the erroneous value among the data from the source database includes:
detecting a value from the data received from the source database that transgresses one or more threshold values from among the set of threshold value.
5. The method of claim 1 , wherein the accessing the set of expected values based on the source database includes:
receiving an input that assigns the set of expected values to an identifier that identifies the source database; and
accessing the set of expected values based on the identifier of the source database.
6. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
receiving, at a local database, data from a source database among a plurality of databases;
determining the source database of the data from among the plurality of databases based on an attribute of the data;
accessing historical data associated with the source database of the data, the historical data comprising a set of expected values that correspond with the source database of the data;
determining a threshold value based on the set of expected values of the historical data associated with the source database;
performing a comparison of the data from the source database with the threshold value;
detecting an erroneous value among the data from the source database based on the comparison of the data from the source database with the threshold value;
causing display of a notification that indicates the erroneous value at a client device in response to the detecting the erroneous value; and
generating a report that comprises a display of at least a portion of the data from the source database.
7. The non-transitory machine-readable storage medium of claim 6 , wherein the data from the source database comprises a data type, and the accessing the set of expected values includes:
accessing the set of expected values based on the source database and the data type of the data.
8. The non-transitory machine-readable storage medium of claim 6 , wherein the set of expected values include a set of threshold values, and the detecting the erroneous value among the data from the source database includes:
detecting a value from the data received from the source database that transgresses one or more threshold values from among the set of threshold value.
9. The non-transitory machine-readable storage medium of claim 6 , wherein the accessing the set of expected values based on the source database includes:
receiving an input that assigns the set of expected values to an identifier that identifies the source database; and
accessing the set of expected values based on the identifier of the source database.
10. A system comprising:
processors; and
a memory storing instructions that, when executed by at least one processor among the processors, cause the system to perform operations comprising:
receiving, at a local database, data from a source database among a plurality of databases;
determining the source database of the data from among the plurality of databases based on an attribute of the data;
accessing historical data associated with the source database of the data, the historical data comprising a set of expected values that correspond with the source database of the data;
determining a threshold value based on the set of expected values of the historical data associated with the source database;
performing a comparison of the data from the source database with the threshold value;
detecting an erroneous value among the data from the source database based on the comparison of the data from the source database with the threshold value;
causing display of a notification that indicates the erroneous value at a client device in response to the detecting the erroneous value; and
generating a report that comprises a display of at least a portion of the data from the source database.
11. The system of claim 10 , wherein the display of the portion of the data from the source database includes a presentation of the erroneous value, wherein the presentation of the erroneous value is visually distinguished from the display of the data.
12. The system of claim 10 , wherein the data from the source database comprises a data type, and the accessing the set of expected values includes:
accessing the set of expected values based on the source database and the data type of the data.
13. The system of claim 10 , wherein the set of expected values include a set of threshold values, and the detecting the erroneous value among the data from the source database includes:
detecting a value from the data received from the source database that transgresses one or more threshold values from among the set of threshold value.
14. The system of claim 10 , wherein the accessing the set of expected values based on the source database includes:
receiving an input that assigns the set of expected values to an identifier that identifies the source database; and
accessing the set of expected values based on the identifier of the source database.