Output validation of data processing systems
A method is provided for output validation of data processing systems, performed by one or more processors. The method comprises performing a data comparison between a first data table and a second data table to determine a data differentiating table, wherein the first data table is based on an output of a first data pipeline, and wherein the second data table is based on an output of a second data pipeline; performing a schema comparison between the first data table and the second data table to determine a schema differentiating table; generating a first output validation score based on the data differentiating table; generating a second output validation score based on the schema differentiating table; and generating a summary comprising both the first and second output validation scores.
1. A computer-implemented method comprising:
generating a first aggregated data table based on an output of a first data pipeline, wherein generating the first aggregated data table includes performing aggregations on the output of the first data pipeline to determine aggregation values, wherein performing aggregations on the output of the first data pipeline includes calculating first hash values for respective columns or rows of the output of the first data pipeline, and wherein the aggregation values of the first aggregated data table include the calculated first hash values;
generating a second aggregated data table based on an output of a second data pipeline, wherein generating the second aggregated data table performing aggregations on the output of the second data pipeline to determine aggregation values, wherein performing aggregations on the output of the second data pipeline includes calculating second hash values for respective columns or rows of the output of the second data pipeline, and wherein the aggregation values of the second aggregated data table include the calculated second hash values;
performing a data comparison between the first aggregated data table and the second aggregated data table to determine a data differentiating table, wherein the data comparison includes comparing respective pairs of the calculated first hash values and the calculated second hash values;
performing a schema comparison between the first aggregated data table and the second aggregated data table to determine a schema differentiating table;
generating a first output validation score based on the data differentiating table;
generating a second output validation score based on the schema differentiating table; and
generating a summary comprising both the first and second output validation scores.
2. The computer-implemented method of claim 1 , wherein the first and second aggregated data tables comprise aggregations of the outputs of the respective first and second data pipelines.
3. The computer-implemented method of claim 1 , wherein the aggregation values of the first and second aggregated data tables are determined by at least one of: summing, averaging, determining a median, determining a minimum, determining a maximum, determining a variance, determining a kurtosis, determining a standard deviation for numeric values, calculating a hash value for concatenated string values, calculating hash values for numeric values, or using a histogram of characters in a string value.
4. The computer-implemented method of claim 1 , wherein the first and second aggregated data tables include aggregation values in each of the rows of the first and second aggregated data tables, and wherein generating the first and second aggregated data tables includes performing aggregations column-wise or row-wise on the outputs of the respective first and second data pipelines to obtain one aggregation value per column or row, respectively, of the outputs of the respective first and second data pipelines.
5. The computer-implemented method of claim 1 , wherein the data differentiating table includes indications of, for each of the columns of the first and second aggregated data tables, a plurality of data comparison characteristics.
6. The computer-implemented method of claim 1 , wherein the schema differentiating table includes indications of, for each of the columns of the first and second aggregated data tables, a plurality of schema comparison characteristics.
7. The computer-implemented method of claim 1 , wherein the first data pipeline is a legacy data pipeline and the second data pipeline is a target data pipeline that is intended to replace the first data pipeline, and wherein the first aggregated data table is transferred to a second data processing system associated with the second data pipeline, and wherein the data comparison and the schema comparison are performed at the second data processing system.
8. The computer-implemented method of claim 1 , wherein the schema differentiating table includes indications of at least: names of each column in the first aggregated data table, any columns present in the first aggregated data table but not in the second aggregated data table, and any columns with a type mismatch between the first and second aggregated data tables.
9. The computer-implemented method of claim 8 , wherein the schema comparison is based on a comparison of datatypes and/or names of one or more columns of the first aggregated data table and the second aggregated data table.
10. The computer-implemented method of claim 1 , wherein the generating the summary comprises using weights to obtain a use case aware summary, and wherein the summary indicates a similarity between the output of the first data pipeline and the output of the second data pipeline.
11. The computer-implemented method of claim 10 , wherein at least one of: a user can determine how columns and/or rows are to be weighted; or weights of columns and/or rows are automatically determined based on user input in one or more dashboards.
12. The computer-implemented method of claim 1 further comprising:
causing display of the summary as a summary table.
13. A system comprising one or more processors and memory storing instructions that, when executed by the one or more processors, cause the system to:
generate a first aggregated data table based on an output of a first data pipeline, wherein generating the first aggregated data table includes performing aggregations on the output of the first data pipeline to determine aggregation values, wherein performing aggregations on the output of the first data pipeline includes calculating first hash values for respective columns or rows of the output of the first data pipeline, and wherein the aggregation values of the first aggregated data table include the calculated first hash values;
generate a second aggregated data table based on an output of a second data pipeline, wherein generating the second aggregated data table performing aggregations on the output of the second data pipeline to determine aggregation values, wherein performing aggregations on the output of the second data pipeline includes calculating second hash values for respective columns or rows of the output of the second data pipeline, and wherein the aggregation values of the second aggregated data table include the calculated second hash values;
perform a data comparison between the first aggregated data table and the second aggregated data table to determine a data differentiating table, wherein the data comparison includes comparing respective pairs of the calculated first hash values and the calculated second hash values;
perform a schema comparison between the first aggregated data table and the second aggregated data table to determine a schema differentiating table;
generate a first output validation score based on the data differentiating table;
generate a second output validation score based on the schema differentiating table; and
generate a summary comprising both the first and second output validation scores.
14. The system of claim 13 , wherein the first and second aggregated data tables comprise aggregations of the outputs of the respective first and second data pipelines.
15. The system of claim 13 , wherein the aggregation values of the first and second aggregated data tables are determined by at least one of: summing, averaging, determining a median, determining a minimum, determining a maximum, determining a variance, determining a kurtosis, determining a standard deviation for numeric values, calculating a hash value for concatenated string values, calculating hash values for numeric values, or using a histogram of characters in a string value.
16. The system of claim 13 , wherein the first and second aggregated data tables include aggregation values in each of the rows of the first and second aggregated data tables, and wherein generating the first and second aggregated data tables includes performing aggregations column-wise or row-wise on the outputs of the respective first and second data pipelines to obtain one aggregation value per column or row, respectively, of the outputs of the respective first and second data pipelines.
17. The system of claim 13 , wherein the data differentiating table includes indications of, for each of the columns of the first and second aggregated data tables, a plurality of data comparison characteristics.
18. The system of claim 13 , wherein the schema differentiating table includes indications of, for each of the columns of the first and second aggregated data tables, a plurality of schema comparison characteristics.
19. The system of claim 13 , wherein the first data pipeline is a legacy data pipeline and the second data pipeline is a target data pipeline that is intended to replace the first data pipeline, and wherein the first aggregated data table is transferred to a second data processing system associated with the second data pipeline, and wherein the data comparison and the schema comparison are performed at the second data processing system.
20. One or more non-transitory computer-readable mediums comprising computer executable instructions stored thereon which, when executed by one or more processors, cause the one or more processors to:
generate a first aggregated data table based on an output of a first data pipeline, wherein generating the first aggregated data table includes performing aggregations on the output of the first data pipeline to determine aggregation values, wherein performing aggregations on the output of the first data pipeline includes calculating first hash values for respective columns or rows of the output of the first data pipeline, and wherein the aggregation values of the first aggregated data table include the calculated first hash values;
generate a second aggregated data table based on an output of a second data pipeline, wherein generating the second aggregated data table performing aggregations on the output of the second data pipeline to determine aggregation values, wherein performing aggregations on the output of the second data pipeline includes calculating second hash values for respective columns or rows of the output of the second data pipeline, and wherein the aggregation values of the second aggregated data table include the calculated second hash values;
perform a data comparison between the first aggregated data table and the second aggregated data table to determine a data differentiating table, wherein the data comparison includes comparing respective pairs of the calculated first hash values and the calculated second hash values;
perform a schema comparison between the first aggregated data table and the second aggregated data table to determine a schema differentiating table;
generate a first output validation score based on the data differentiating table;
generate a second output validation score based on the schema differentiating table; and
generate a summary comprising both the first and second output validation scores.