Systems and methods for measuring data quality over time
Systems, methods, and computer-readable media are disclosed for evaluating data quality. An exemplary embodiment includes storing a plurality of records, the records sharing a common attribute, and reading first values for the common attribute corresponding to a first time period and second values for the common attribute corresponding to a second time period. A business rule for evaluating the common attribute is accessed, and first and second consistency data are generated. The first consistency data may reflect the extent to which the first values of the common attribute are consistent with the business rule at the first time. The second consistency data may reflect the extent to which the second values of the common attribute are consistent with the business rule at the second time. The first consistency data and the second consistency data are processed to generate a quality change rate of the common attribute from the first time period to the second time period, based on the difference between the first consistency data and the second consistency data.
1 . A computer-implemented method, performed by a computing platform connected to a database and a processor, for determining a relationship between data volatility and data quality of attributes of electronic business records, comprising:
reading, by the processor and from the database, a first data quality metric for a first data set corresponding to a first time period, wherein the first data quality metric is based at least in part on first consistency data for the first data set derived from evaluating, by the processor, first values of a common attribute corresponding to a first time period against a business rule for evaluating the common attribute;
reading, by the processor and from the database, a first data volatility metric for the first data set corresponding to the first time period;
reading, by the processor and from the database, a second data volatility metric for the first data set corresponding to a second time period;
determining, by the processor, a relationship between the first data volatility metric and the second data volatility metric;
determining, by the processor, a second data quality metric for the first data set corresponding to the second time period based on the determined relationship and
determining, by the processor, a predicted future outcome corresponding to the second time period by fitting a curve to data reflecting a second relationship between the first data quality metric for the first data set and outcomes of one or more electronic business records corresponding to the first data set and mapping the determined second data quality metric for the first data set to the predicted future outcome,
wherein the first data quality metric and the second data quality metric correspond to a same data quality metric in two different time periods, and
wherein the first data volatility metric and the second data volatility metric correspond to a same data volatility metric in two different time periods.
2 . The computer-implemented method of claim 1 , wherein determining the relationship between the first data volatility metric and the second data volatility metric is based on a detected change between the first time period and the second time period.
3 . The computer-implemented method of claim 1 , wherein determining the second data quality metric comprises positively correlating the second data quality metric and the determined relationship based on a type of the first data set.
4 . The computer-implemented method of claim 1 , wherein determining the second data quality metric comprises negatively correlating the second data quality metric and the determined relationship based on a type of the first data set.
5 . The computer-implemented method of claim 1 , further comprising:
determining, by the processor, a source of an error associated with the first data set based on the second data quality metric.
6 . The computer-implemented method of claim 1 , further comprising:
determining, by the processor, a third data quality metric for a second data set based on the first data quality metric or the second data quality metric.
7 . The computer-implemented method of claim 1 , further comprising:
determining, by the processor, a third data volatility metric for a second data set based on the first data volatility metric or the second data volatility metric.
8 . A computing system for determining a relationship between data volatility and data quality of attributes of electronic business records, the system comprising:
a memory configured to store instructions;
a database; and
a processor configured to execute the stored instructions to perform operations comprising:
reading, by the processor and from the database, a first data quality metric for a first data set corresponding to a first time period, wherein the first data quality metric is based at least in part on first consistency data for the first data set derived from evaluating, by the processor, first values of a common attribute corresponding to a first time period against a business rule for evaluating the common attribute;
reading, by the processor and from the database, a first data volatility metric for the first data set corresponding to the first time period;
reading, by the processor and from the database, a second data volatility metric for the first data set corresponding to a second time period;
determining, by the processor, a relationship between the first data volatility metric and the second data volatility metric;
determining, by the processor, a second data quality metric for the first data set corresponding to the second time period based on the determined relationship; and
determining, by the processor, a predicted future outcome corresponding to the second time period by fitting a curve to data reflecting a second relationship between the first data quality metric for the first data set and outcomes of one or more electronic business records corresponding to the first data set and mapping the determined second data quality metric for the first data set to the predicted future outcome,
wherein the first data quality metric and the second data quality metric correspond to a same data quality metric in two different time periods, and
wherein the first data volatility metric and the second data volatility metric correspond to a same data volatility metric in two different time periods.
9 . The system of claim 8 , wherein determining the relationship between the first data volatility metric and the second data volatility metric is based on a detected change between the first time period and the second time period.
10 . The system of claim 8 , wherein determining the second data quality metric comprises positively correlating the second data quality metric and the determined relationship based on a type of the first data set.
11 . The system of claim 8 , wherein determining the second data quality metric comprises negatively correlating the second data quality metric and the determined relationship based on a type of the first data set.
12 . The system of claim 8 , wherein the operations further comprise:
determining, by the processor, a source of an error associated with the first data set based on the second data quality metric.
13 . The system of claim 8 , wherein the operations further comprise:
determining, by the processor, a third data quality metric for a second data set based on the first data quality metric or the second data quality metric.
14 . The system of claim 8 , wherein the instructions operations further comprise:
determining, by the processor, a third data volatility metric for a second data set based on the first data volatility metric or the second data volatility metric.
15 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor connected to a computing platform and a database, cause the processor to perform operations for determining a relationship between data volatility and data quality of attributes of electronic business records, the operations comprising:
reading, from the database, a first data quality metric for a first data set corresponding to a first time period, wherein the first data quality metric is based at least in part on first consistency data for the first data set derived from evaluating, by the processor, first values of a common attribute corresponding to a first time period against a business rule for evaluating the common attribute;
reading, from the database, a first data volatility metric for the first data set corresponding to the first time period;
reading, from the database, a second data volatility metric for the first data set corresponding to a second time period;
determining a relationship between the first data volatility metric and the second data volatility metric;
determining a second data quality metric for the first data set corresponding to the second time period based on the determined relationship; and
determining, by the processor, a predicted future outcome corresponding to the second time period by fitting a curve to data reflecting a second relationship between the first data quality metric for the first data set and outcomes of one or more electronic business records corresponding to the first data set and mapping the determined second data quality metric for the first data set to the predicted future outcome,
wherein the first data quality metric and the second data quality metric correspond to a same data quality metric in two different time periods, and
wherein the first data volatility metric and the second data volatility metric correspond to a same data volatility metric in two different time periods.
16 . The non-transitory computer-readable storage medium of claim 15 , wherein determining the relationship between the first data volatility metric and the second data volatility metric is based on a detected change between the first time period and the second time period.
17 . The non-transitory computer-readable storage medium of claim 15 , wherein determining the second data quality metric comprises positively correlating the second data quality metric and the determined relationship based on a type of the first data set.
18 . The non-transitory computer-readable storage medium of claim 15 , wherein determining the second data quality metric comprises negatively correlating the second data quality metric and the determined relationship based on a type of the first data set.
19 . The non-transitory computer-readable storage medium of claim 15 , wherein the operations further comprise:
determining a source of an error associated with the first data set based on the second data quality metric.
20 . The non-transitory computer-readable storage medium of claim 15 , wherein the operations further comprise:
determining a third data quality metric for a second data set based on the first data quality metric or the second data quality metric; and
determining a third data volatility metric for a second data set based on the first data volatility metric or the second data volatility metric.