Automated dynamic data quality assessment
View Patent ↗In general, embodiments of the present invention provide systems, methods and computer readable media for automated dynamic data quality assessment. One aspect of the subject matter described in this specification includes the actions of receiving a data quality job including a new data sample; and, if the new data sample is determined to be added to a reservoir of data samples, sending a quality verification request to an oracle; receiving a new data sample quality estimate from the oracle; and adding the new data sample and estimate to the reservoir. A second aspect of the subject matter includes the actions of receiving, from a predictive model, a judgment associated with a new data sample; analyzing the new data sample based in part on the judgment to determine whether to send a new data sample quality verification request to an oracle; and, if a new data sample quality estimate is received from the oracle, determining whether to add the new data sample and the judgment to the reservoir.
1. A computer-implemented method, comprising:
receiving, by a processor, a data quality job, the data quality job including configuration data and a new data sample having a particular data type and one or more attributes associated therewith;
determining, by the processor, whether to add the new data sample to a reservoir of data samples, the reservoir of data samples identified based at least in part on the particular data type; and
in an instance in which the new data sample is to be added to the reservoir of data samples,
sending, by the processor, a quality verification request including the new data sample to an oracle of a plurality of oracles, the oracle selected based on one or more of the particular data type and the one or more attributes associated with the new data sample;
receiving, by the processor, a data quality estimate associated with the new data sample from the oracle in response to the quality verification request; and
adding, by the processor, the new data sample and the associated data quality estimate to the reservoir of data samples in response to receiving the data quality estimate.
2. The method of claim 1 , further comprising:
updating, by the processor, the reservoir summary statistics.
3. The method of claim 2 , wherein updating, by the processor, the reservoir summary statistics comprises:
calculating an overall data quality estimate for the reservoir using data quality estimates respectively associated with each of the data samples; and
calculating a statistical variance for the data samples.
4. The method of claim 2 , wherein updating, by the processor, the reservoir summary statistics further comprises:
logging the updated reservoir summary statistics in persistent storage.
5. The method of claim 2 , further comprising:
receiving, by the processor, corpus summary statistics calculated for a corpus of previously collected data samples, wherein each of the previously collected data samples are respectively associated with the particular data type; and
generating, by the processor, an analysis comparing the updated reservoir summary statistics with the corpus summary statistics.
6. The method of claim 1 , wherein determining whether to add the new data sample to the reservoir is based on at the value of at least one of the attributes of the new data sample.
7. The method of claim 1 , wherein determining whether to add the new data sample to the reservoir is based on a probabilistic sampling approach.
8. The method of claim 1 , wherein the oracle is a crowd, a flat file of previously received crowd data verification results, or a software system.
9. The method of claim 1 , wherein the new data sample is collected from a data stream.
10. The method of claim 9 , wherein the new data sample is a single data instance or a set of data instances collected from the data stream within a pre-defined time window.
11. The method of claim 1 , wherein the new data sample has been pre-processed by a data cleaning process.
12. A computer program product, stored on a non-transitory computer readable medium, comprising instructions that when executed on one or more computers cause the one or more computers to:
receive, by a processor, a data quality job, the data quality job including configuration data and a new data sample having a particular data type and one or more attributes associated therewith;
determine, by the processor, whether to add the new data sample to a reservoir of data samples, the reservoir of data samples identified based at least in part on the particular data type; and
in an instance in which the new data sample is to be added to the reservoir of data samples,
send, by the processor, a quality verification request including the new data sample to an oracle of a plurality of oracles, the oracle selected based on one or more of the particular data type and the one or more attributes associated with the new data sample;
receive, by the processor, a data quality estimate associated with the new data sample from the oracle in response to the quality verification request; and
add, by the processor, the new data sample and the associated data quality estimate to the reservoir of data samples in response to receiving the data quality estimate.
13. The computer program product of claim 12 , wherein the instructions that when executed on one or more computers further cause the one or more computers to:
update the reservoir summary statistics.
14. The computer program product of claim 13 , wherein updating the reservoir summary statistics comprises:
calculating an overall data quality estimate for the reservoir using data quality estimates respectively associated with each of the data samples; and
calculating a statistical variance for the data samples.
15. The computer program product of claim 13 , wherein updating the reservoir summary statistics further comprises:
logging the updated reservoir summary statistics in persistent storage.
16. The computer program product of claim 13 , wherein the instructions that when executed on one or more computers further cause the one or more computers to:
receive corpus summary statistics calculated for a corpus of previously collected data samples, wherein each of the previously collected data samples are respectively associated with the particular data type; and
generate an analysis comparing the updated reservoir summary statistics with the corpus summary statistics.
17. The computer program product of claim 12 , wherein determining whether to add the new data sample to the reservoir is based on at the value of at least one of the attributes of the new data sample.
18. The computer program product of claim 12 , wherein determining whether to add the new data sample to the reservoir is based on a probabilistic sampling approach.
19. The computer program product of claim 12 , wherein the oracle is a crowd, a flat file of previously received crowd data verification results, or a software system.
20. The computer program product of claim 12 , wherein the new data sample is collected from a data stream.
21. The computer program product of claim 20 , wherein the new data sample is a single data instance or a set of data instances collected from the data stream within a pre-defined time window.
22. The computer program product of claim 12 , wherein the new data sample has been pre-processed by a data cleaning process.
23. A system, comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to:
receive, by a processor, a data quality job, the data quality job including configuration data and a new data sample having a particular data type and one or more attributes associated therewith;
determine, by the processor, whether to add the new data sample to a reservoir of data samples, the reservoir of data samples identified based at least in part on the particular data type; and
in an instance in which the new data sample is to be added to the reservoir of data samples,
send, by the processor, a quality verification request including the new data sample to an oracle of a plurality of oracles, the oracle selected based on one or more of the particular data type and the one or more attributes associated with the new data sample;
receive, by the processor, a data quality estimate associated with the new data sample from the oracle in response to the quality verification request; and
add, by the processor, the new data sample and the associated data quality estimate to the reservoir of data samples in response to receiving the data quality estimate.
24. The system of claim 23 , wherein the instructions are further operable, when executed by the one or more computers, to cause the one or more computers to:
update the reservoir summary statistics.
25. The system of claim 24 , wherein updating the reservoir summary statistics comprises:
calculating an overall data quality estimate for the reservoir using data quality estimates respectively associated with each of the data samples; and
calculating a statistical variance for the data samples.
26. The system of claim 24 , wherein updating the reservoir summary statistics further comprises:
logging the updated reservoir summary statistics in persistent storage.
27. The system of claim 24 , wherein the instructions are further operable, when executed by the one or more computers, to cause the one or more computers to:
receive corpus summary statistics calculated for a corpus of previously collected data samples, wherein each of the previously collected data samples are respectively associated with the particular data type; and
generate an analysis comparing the updated reservoir summary statistics with the corpus summary statistics.
28. The system of claim 23 , wherein determining whether to add the new data sample to the reservoir is based on at the value of at least one of the attributes of the new data sample.
29. The system of claim 23 , wherein determining whether to add the new data sample to the reservoir is based on a probabilistic sampling approach.
30. The system of claim 23 , wherein the oracle is a crowd, a flat file of previously received crowd data verification results, or a software system.
31. The system of claim 23 , wherein the new data sample is collected from a data stream.
32. The system of claim 31 , wherein the new data sample is a single data instance or a set of data instances collected from the data stream within a pre-defined time window.
33. The system of claim 23 , wherein the new data sample has been pre-processed by a data cleaning process.