System and method for checking data for errors

1. A system for checking data for errors, the system comprising:

a checking module operable to check tuples of data stored in a target database for errors, the tuples in the target database originating from the output of at least one query transformation module which applies a query transformation to tuples of data from at least one data source;

an identification module operable to identify a problematic tuple from a data source that produces an error in the target database, the identification module being operable to quantify the contribution of the problematic tuple in producing the error in the target database,

a description generation module operable to generate a descriptive query which represents at least one of:

errors identified by the checking module in the target database which are produced by the at least one query transformation module, and

problematic tuples identified in a data source by the identification module; and

a correction module which is operable to use the descriptive query to modify at least one of:

the at least one query transformation module to correct an error produced by the at least one query transformation module; and

a data source to correct problematic tuples in the data source.

2. The system of claim 1 , wherein the descriptive query comprises lineage data which indicates at least one of a query transformation module producing the error and a data source comprising a problematic tuple.

3. The system of claim 1 , wherein the system further comprises the at least one transformation module and the at least one transformation module is operable to modify the transformation applied by the transformation module so that the transformation module does not produce an error in the target database.

4. The system of claim 1 , wherein the checking module is operable to receive at least one quality rule and to check the data stored in the target database to detect if the data violates each quality rule, and wherein the system further comprises a violations storage module which is operable to store data that violates at least one of the quality rules in a violation table.

5. The system of claim 1 , wherein the checking module is operable to

receive at least one quality rule and to check the data stored in the target database to detect if the data violates each quality rule and identify at least one attribute in a tuple of data stored in the violation table that violates at least one of the quality rules, and to identify the data source from which the attribute originated; and

wherein the system further comprises a violations storage module which is operable to store data that violates at least one of the quality rules in a violation table.

6. The system of claim 1 , wherein the checking module is operable to

receive at least one quality rule and to check the data stored in the target database to detect if the data violates each quality rule and identify at least one attribute in a tuple of data stored in the violation table that violates at least one of the quality rules, and to identify the data source from which the attribute originated;

wherein the system further comprises a violations storage module which is operable to store data that violates at least one of the quality rules in a violation table; and

wherein a processing module operable to process the data stored in the violations table to identify an error value for at least one attribute in the violations table, the error value indicating the probability of the attribute violating a quality rule.

7. The system of claim 1 , wherein the checking module is operable to

receive at least one quality rule and to check the data stored in the target database to detect if the data violates each quality rule and identify at least one attribute in a tuple of data stored in the violation table that violates at least one of the quality rules, and to identify the data source from which the attribute originated;

wherein the system further comprises a violations storage module which is operable to store data that violates at least one of the quality rules in a violation table;

wherein a processing module is operable to process the data stored in the violations table to identify an error value for at least one attribute in the violations table, the error value indicating the probability of the attribute violating a quality rule; and

wherein the system further comprises a query module which is operable to provide at least one query to the target database and to record the number of clean and erroneous tuples of data that are returned by the at least one query.

8. The system of claim 1 , wherein the checking module is operable to

receive at least one quality rule and to check the data stored in the target database to detect if the data violates each quality rule and identify at least one attribute in a tuple of data stored in the violation table that violates at least one of the quality rules, and to identify the data source from which the attribute originated;

wherein the system further comprises a violations storage module which is operable to store data that violates at least one of the quality rules in a violation table;

wherein a processing module is operable to process the data stored in the violations table to identify an error value for at least one attribute in the violations table, the error value indicating the probability of the attribute violating a quality rule, and to store an annotation associated with the record of each tuple of data stored in the violations table with a weight value indicating the probability of the tuple violating a quality rule in response to a query to the target database; and

wherein the system further comprises a query module which is operable to provide at least one query to the target database and to record the number of clean and erroneous tuples of data that are returned by the at least one query.

9. The system of claim 1 , wherein the checking module is operable to

receive at least one quality rule and to check the data stored in the target database to detect if the data violates each quality rule and identify at least one attribute in a tuple of data stored in the violation table that violates at least one of the quality rules, and to identify the data source from which the attribute originated;

wherein the system further comprises a violations storage module which is operable to store data that violates at least one of the quality rules in a violation table;

wherein a processing module is operable to process the data stored in the violations table to identify an error value for at least one attribute in the violations table, the error value indicating the probability of the attribute violating a quality rule, and to store an annotation associated with the record of each tuple of data stored in the violations table with a weight value indicating the probability of the tuple violating a quality rule in response to a query to the target database;

wherein the system further comprises a query module which is operable to provide at least one query to the target database and to record the number of clean and erroneous tuples of data that are returned by the at least one query; and

wherein the system further comprises a contribution score vector calculation module operable to calculate a contribution score vector indicating the probability of a tuple of data causing an error, and wherein the processing module is operable to annotate the record of each tuple of data stored in the violations table with the calculated contribution score vector.

10. The system of claim 1 , wherein the checking module is operable to

receive at least one quality rule and to check the data stored in the target database to detect if the data violates each quality rule and identify at least one attribute in a tuple of data stored in the violation table that violates at least one of the quality rules, and to identify the data source from which the attribute originated;

wherein the system further comprises a violations storage module which is operable to store data that violates at least one of the quality rules in a violation table;

wherein a processing module is operable to process the data stored in the violations table to identify an error value for at least one attribute in the violations table, the error value indicating the probability of the attribute violating a quality rule, and to store an annotation associated with the record of each tuple of data stored in the violations table with a weight value indicating the probability of the tuple violating a quality rule in response to a query to the target database;

wherein the system further comprises a query module which is operable to provide at least one query to the target database and to record the number of clean and erroneous tuples of data that are returned by the at least one query;

wherein the system further comprises a contribution score vector calculation module operable to calculate a contribution score vector indicating the probability of a tuple of data causing an error; and wherein the processing module is operable to annotate the record of each tuple of data stored in the violations table with the calculated contribution score vector; and

wherein the system further comprises a removal score vector calculation module operable to calculate a removal score vector which indicates if a violation can be removed by removing a tuple of data from a data source.

11. The system of claim 1 , wherein the checking module is operable to

receive at least one quality rule and to check the data stored in the target database to detect if the data violates each quality rule and identify at least one attribute in a tuple of data stored in the violation table that violates at least one of the quality rules, and to identify the data source from which the attribute originated;

wherein the system further comprises a violations storage module which is operable to store data that violates at least one of the quality rules in a violation table;

wherein a processing module is operable to process the data stored in the violations table to identify an error value for at least one attribute in the violations table, the error value indicating the probability of the attribute violating a quality rule, and to store an annotation associated with the record of each tuple of data stored in the violations table with a weight value indicating the probability of the tuple violating a quality rule in response to a query to the target database;

wherein the system further comprises a query module which is operable to provide at least one query to the target database and to record the number of clean and erroneous tuples of data that are returned by the at least one query;

wherein the system further comprises a contribution score vector calculation module operable to calculate a contribution score vector indicating the probability of a tuple of data causing an error, and wherein the processing module is operable to annotate the record of each tuple of data stored in the violations table with the calculated contribution score vector; and

wherein the system further comprises a distance calculation module operable to calculate the relative distance between the tuples in the data entries stored in the violations table that have a contribution score vector or a removal score vector above a predetermined threshold.

12. A computer implemented method for checking data for errors, the method comprising:

checking tuples of data stored in a target database for errors, the tuples in the target database originating from the output of at least one query transformation module which applies a query transformation to tuples of data from at least one data source;

identifying a problematic tuple from a data source that produces an error in the target database and quantifying the contribution of the problematic tuple in producing the error in the target database,

generating a descriptive query which represents at least one of;

errors identified by the checking of the tuples of data stored in the target database which are produced by the at least one query transformation module, and

problematic tuples identified in a data source by the identification of the problematic tuple from the data source, and

using the descriptive query to modify at least one of:

the at least one query transformation module to correct an error produced by the at least one query transformation module; and

a data source to correct problematic tuples in the data source.

13. The method of claim 12 ,

using the descriptive query to modify at least one of:

the at least one query transformation module to correct an error produced by the at least one query transformation module; and

a data source to correct problematic tuples in the data source; and

wherein the descriptive query comprises lineage data which indicates at least one of a query transformation module producing the error and a data source comprising a problematic tuple.

14. The method of claim 12 , wherein the checking step comprises:

providing at least one quality rule;

checking the data stored in the target database to detect if the data violates each quality rule; and

storing the data that violates at least one of the quality rules in a violation table.

15. The method of claim 12 , wherein the checking step comprises:

providing at ea one quality rule;

checking the data stored in the target database to detect if the data violates each quality rule;

storing the data that violates at least one of the quality rules in a violation table;

identifying at least one attribute in a tuple of data stored in the violation table that violates at least one of the quality rules; and

identifying the data source from which the attribute originated.

16. The method of claim 12 , wherein the checking step comprises:

providing at least one quality rule;

checking the data stored in the target database to detect if the data violates each quality rule;

storing the data that violates at least one of the quality rules in a violation table;

identifying at least one attribute in a tuple of data stored in the violation table that violates at least one of the quality rules;

identifying the data source from which the attribute originated; and

processing the data stored in the violations table to identify an error value for at least one attribute in the violations table, the error value indicating the probability of the attribute violating a quality rule.

17. The method of claim 12 , wherein the checking step comprises:

providing at least one quality rule;

checking the data stored in the target database to detect if the data violates each quality rule;

storing the data that violates at least one of the quality rules in a violation table;

identifying at least one attribute in a tuple of data stored in the violation table that violates at least one of the quality rules;

identifying the data source from which the attribute originated; and

processing the data stored in the violations table to identify an error value for at least one attribute in the violations table, the error value indicating the probability of the attribute violating a quality rule; and

providing at least one query to the target database and recording the number of clean and erroneous tuples of data that are returned by the at least one query.

18. The method of claim 12 , wherein the checking step comprises:

providing at least one quality rule;

checking the data stored in the target database to detect if the data violates each quality rule;

storing the data that violates at least one of the quality rules in a violation table;

identifying at least one attribute in a tuple of data stored in the violation table that violates at least one of the quality rules;

identifying the data source from which the attribute originated;

processing the data stored in the violations table to identify an error value for at least one attribute in the violations table, the error value indicating the probability of the attribute violating a quality rule;

providing at least one query to the target database and recording the number of clean and erroneous tuples of data that are returned by the at least one query; and

annotating the record of each tuple of data stored in the violations table with a weight value indicating the likelihood of the tuple violating a quality rule in response to a query to the target database.

19. The method of claim 12 , wherein the checking step comprises:

providing at least one quality rule;

checking the data stored in the target database to detect if the data violates each quality rule;

storing the data that violates at least one of the quality rules in a violation table;

identifying at least one attribute in a tuple of data stored in the violation table that violates at least one of the quality rules;

identifying the data source from which the attribute originated; and

processing the data stored in the violations table to identify an error value for at least one attribute in the violations table, the error value indicating the probability of the attribute violating a quality rule;

providing at least one query to the target database and recording the number of clean and erroneous tuples of data that are returned by the at least one query;

annotating the record of each tuple of data stored in the violations table with a weight value indicating the likelihood of the tuple violating a quality rule in response to a query to the target database; and

calculating a contribution score vector indicating the probability of a tuple of data causing an error and annotating the record of each tuple of data stored in the violations table with the calculated contribution score vector.

20. The method of claim 12 , wherein the checking step comprises:

providing at least one quality rule;

checking the data stored in the target database to detect if the data violates each quality rule;

storing the data that violates at least one of the quality rules in a violation table;

identifying at least one attribute in a tuple of data stored in the violation table that violates at least one of the quality rules;

identifying the data source from which the attribute originated;

processing the data stored in the violations table to identify an error value for at least one attribute in the violations table, the error value indicating the probability of the attribute violating a quality rule;

providing at least one query to the target database and recording the number of clean and erroneous tuples of data that are returned by the at least one query;

annotating the record of each tuple of data stored in the violations table with a weight value indicating the likelihood of the tuple violating a quality rule in response to a query to the target database;

calculating a contribution score vector indicating the probability of a tuple of data causing an error and annotating the record of each tuple of data stored in the violations table with the calculated contribution score vector; and

computing a removal score vector which indicates if a violation can be removed by removing a tuple of data from a data source.

21. The method of claim 12 , wherein the checking step comprises:

providing at least one quality rule;

checking the data stored in the target database to detect if the data violates each quality rule;

storing the data that violates at least one of the quality rules in a violation table;

identifying at least one attribute in a tuple of data stored in the violation table that violates at least one of the quality rules;

identifying the data source from which the attribute originated; and

processing the data stored in the violations table to identify an error value for at least one attribute in the violations table, the error value indicating the probability of the attribute violating a quality rule;

providing at least one query to the target database and recording the number of clean and erroneous tuples of data that are returned by the at least one query;

annotating the record of each tuple of data stored in the violations table with a weight value indicating the likelihood of the tuple violating a quality rule in response to a query to the target database;

calculating a contribution score vector indicating the probability of a tuple of data causing an error and annotating the record of each tuple of data stored in the violations table with the calculated contribution score vector; computing a removal score vector which indicates if a violation can be removed by removing a tuple of data from a data source; and

determining the relative distance between the tuples in the data entries stored in the violations table that have a contribution score vector or a removal score vector above a predetermined threshold.