Systems and methods for validating data
Systems and methods are validating data in a data set. A data set including data to validate and a validator to use in validating the data is selected based on user input generated based on interactions of a user with a graphical user interface. The validator is applied to the data to determine whether one or more statistics generated through application of the validator to the data is valid or invalid based on a validation routine associated with the validator. A data quality report indicating whether the data set is valid or invalid, based on a determination of whether the one or more statistics is valid or invalid, is generated and selectively presented to the user through the graphical user interface.
1 . A system for validating data comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the system to:
obtaining a data set;
detecting that the data set has been modified, where the modified data set comprises additional entries added to the data set, wherein the additional entries result in a changed schema or a new schema of the modified data set compared to the data set;
in response to detecting the additional entries, automatically configuring a new validator based on the additional entries, the new validator being configured for the changed schema or the new schema, wherein the new validator has a new validation parameter or a new validation routine that is distinct from an existing validation parameter or an existing validation routine of an existing validator, and the new validator has the new validation parameter, the new validation routine, a different validation parameter, or a different validation routine that is in common with the existing validation parameter, the existing validation routine, a different existing validation parameter, or a different existing validation routine of the existing validator;
applying the new validator to determine a validity of the modified data set, wherein the validity of the data set is determined based on an evaluation of one or more statistics associated with the data set with respect to the new validation parameter or the new validation routine;
generating one or more statistics associated with the data set; and
providing the statistics to a user.
2 . The system of claim 1 , wherein the application of the new validator comprises determining the validity of the modified data set based on a proportion of entries in the modified data set having null values.
3 . The system of claim 1 , wherein the application of the validator comprises determining the validity of the modified data set based on a validity of join keys among different tables.
4 . The system of claim 1 , wherein the application of the validator comprises determining the validity of the modified data set based on a proportion of rows among different tables that joined or that failed to join.
5 . The system of claim 1 , wherein the validation parameter is distinct from the existing validation parameter and the validation routine is distinct from the existing validation routine.
6 . The system of claim 1 , wherein the application of the validator comprises determining the validity of the modified data set based on schemas of different tables in the modified data set.
7 . The system of claim 1 , wherein the application of the validator comprises determining the validity of the modified data set in response to respective schemas of different tables in the modified data set matching.
8 . The system of claim 1 , wherein the instructions further cause the system to:
perform a validation of the modified data set based on an existing validator prior to applying the new validator.
9 . A method being implemented by a computing system including one or more physical processors and storage media storing machine-readable instructions, the method comprising:
obtaining a data set;
detecting that the data set has been modified, where the modified data set comprises additional entries added to the data set, wherein the additional entries result in a changed schema or a new schema of the modified data set compared to the data set;
in response to detecting the additional entries, automatically configuring a new validator based on the additional entries, the new validator being configured for the changed schema or the new schema, wherein the new validator has a new validation parameter or a new validation routine that is distinct from an existing validation parameter or an existing validation routine of an existing validator, and the new validator has the new validation parameter, the new validation routine, a different validation parameter, or a different validation routine that is in common with the existing validation parameter, the existing validation routine, a different existing validation parameter, or a different existing validation routine of the existing validator;
applying the new validator to determine a validity of the modified data set, wherein the validity of the data set is determined based on an evaluation of one or more statistics associated with the data set with respect to the new validation parameter or the new validation routine;
generating one or more statistics associated with the data set; and
providing the statistics to a user.
10 . The method of claim 9 , wherein the application of the new validator comprises determining the validity of the modified data set based on a proportion of entries in the modified data set having null values.
11 . The method of claim 9 , wherein the application of the validator comprises determining the validity of the modified data set based on a validity of join keys among different tables.
12 . The method of claim 9 , wherein the application of the validator comprises determining the validity of the modified data set based on a proportion of rows among different tables that joined or that failed to join.
13 . The method of claim 9 , wherein the validation parameter is distinct from the existing validation parameter and the validation routine is distinct from the existing validation routine.
14 . The method of claim 9 , wherein the application of the validator comprises determining the validity of the modified data set based on schemas of different tables in the modified data set.
15 . The method of claim 9 , wherein the application of the validator comprises determining the validity of the modified data set in response to respective schemas of different tables in the modified data set matching.
16 . The method of claim 9 , wherein the application of the validator comprises determining the validity of the modified data set based on a row count among different tables in the data set.
17 . The method of claim 9 , further comprising:
in response to applying the validator, determining that the modified data set is invalid; and
in response to determining that the modified data set is invalid, generating an identification of particular statistics that were invalid.
18 . A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:
obtaining a data set;
detecting that the data set has been modified, where the modified data set comprises additional entries added to the data set, wherein the additional entries result in a changed schema or a new schema of the modified data set compared to the data set;
in response to detecting the additional entries, automatically configuring a new validator based on the additional entries, the new validator being configured for the changed schema or the new schema, wherein the new validator has a new validation parameter or a new validation routine that is distinct from an existing validation parameter or an existing validation routine of an existing validator, and the new validator has the new validation parameter, the new validation routine, a different validation parameter, or a different validation routine that is in common with the existing validation parameter, the existing validation routine, a different existing validation parameter, or a different existing validation routine of the existing validator;
applying the new validator to determine a validity of the modified data set, wherein the validity of the data set is determined based on an evaluation of one or more statistics associated with the data set with respect to the new validation parameter or the new validation routine;
generating one or more statistics associated with the data set; and
providing the statistics to a user.
19 . The non-transitory computer readable medium of claim 18 , wherein the application of the new validator comprises determining the validity of the modified data set in response to schemas of different tables in the modified data set matching.