System and method for data quality assessment
Methods and systems for providing data assessment across related datasets, including identifying exceptional values in datasets and assessing upstream and or downstream datasets that utilize the exceptional values. Data assessment rules use exceptional values found in a dataset and data lineage information to identify impacted data in upstream or downstream datasets.
1. A non-transitory computer-readable medium storing instructions for causing a computing system to:
assess one or more first attributes in one or more first records stored in a first dataset;
identify a first attribute with an exceptional value;
store a first entry in an exception table comprising a first object identifier value identifying the first dataset, a first record identifier identifying the first record containing the first attribute with an exceptional value, a first attribute identifier identifying the first attribute with an exceptional value, and a first exception identifier associated with the first entry;
identify, in at least one second dataset, a second record containing a second attribute that utilizes the first attribute with the exceptional value;
store a second entry in the exception table comprising a second record identifier identifying the second record; and
link the second entry to the first entry using the first exception identifier.
2. The medium of claim 1 , wherein the instructions to identify, in at least one second dataset, a second record further comprise instructions to search the second dataset for a record containing at least one of the first record identifier, the first object identifier, and the first attribute identifier.
3. The medium of claim 2 , wherein the instructions to identify, in at least one second dataset, a second record, further comprise instructions to identify a match between the first record identifier in the first entry in the exception table to a second record identifier in the second record.
4. The medium of claim 1 , wherein the instructions to identify, in at least one second dataset, a second record, further comprise instructions to identify a transform function that utilizes the first attribute with the exceptional value to create the second attribute contained in the at least one second record.
5. The medium of claim 1 , further comprising instructions to identify, in at least one third database, a third record that utilizes at least one of the first attribute with the exceptional value and the second attribute.
6. The medium of claim 1 , wherein the instructions to store a second entry in the exception table further comprise instructions to store a second attribute identifier identifying the second attribute and a second object identifier identifying the second dataset.
7. The medium of claim 1 , further comprising instructions to communicate the second record identifier to an entity able to adjust a downstream process utilizing the second record.
8. A computer system, comprising:
one or more processors; and
a non-transitory computer readable medium storing instructions that cause the one or more processors to:
assess one or more first attributes in one or more first records stored in a first dataset;
identify a first attribute with an exceptional value;
store a first entry in an exception table comprising a first object identifier value identifying the first dataset, a first record identifier identifying the first record containing the first attribute with an exceptional value, a first attribute identifier identifying the first attribute with an exceptional value, and a first exception identifier associated with the first entry;
identify, in at least one second dataset, a second record containing a second attribute that utilizes the first attribute with the exceptional value;
store a second entry in the exception table comprising a second record identifier identifying the second record; and
link the second entry to the first entry using the first exception identifier.
9. The computer system of claim 8 , wherein the instructions to identify, in at least one second dataset, a second record further comprise instructions to search the second dataset for a record containing at least one of the first record identifier, the first object identifier, and the first attribute identifier.
10. The computer system of claim 9 , wherein the instructions to identify, in at least one second dataset, a second record, further comprise instructions to identify a match between the first record identifier in the first entry in the exception table to a second record identifier in the second record.
11. The computer system of claim 8 , wherein the instructions to identify, in at least one second dataset, a second record, further comprise instructions to identify a transform function that utilizes the first attribute with the exceptional value to create the second attribute contained in the at least one second record.
12. The computer system of claim 8 , further comprising instructions to identify, in at least one third database, a third record that utilizes at least one of the first attribute with the exceptional value and the second attribute.
13. The computer system of claim 8 , wherein the instructions to store a second entry in the exception table further comprise instructions to store a second attribute identifier identifying the second attribute and a second object identifier identifying the second dataset.
14. The computer system of claim 8 , further comprising instructions to communicate the second record identifier to an entity able to adjust a downstream process utilizing the second record.