System and method for validating data
A system and method are provided for validating data. The method is executed by a device having a data interface coupled to a processor and includes obtaining a validation set comprising at least one validation case, each validation case comprising at least one test condition. The method also includes obtaining, via the data interface, at least one data set to be validated using the validation set. The method also includes applying the validation set to the at least one data set to validate the data in the data set by, for each record in the at least one data set, validating a value in the record according to the at least one test condition. The method also includes outputting a validation result for each record.
1. A device for validating data, the device comprising:
a processor;
a data interface coupled to the processor; and
a memory coupled to the processor, the memory storing computer executable instructions that when executed by the processor cause the device to:
obtain, via the data interface, at least one data set to be validated using a validation set determined according to at least one test condition, wherein the at least one dataset is an output of at least one statistical analysis on at least one input data set;
apply the validation set to the at least one data set to validate the data in the data set by, for at least one record in the at least one data set, validating the record according to the at least one test condition; and
output a validation result for the data set.
2. The device of claim 1 , wherein the computer executable instructions cause the device to:
in response to the validation result satisfying at least one criterion, transmit the at least one data set to a process for consumption.
3. The device of claim 1 , wherein the validation set comprises at least one validation case, each validation case comprising at least one test condition.
4. The device of claim 1 , wherein the computer executable instructions cause the device to:
transmit for adjusting, or adjust, the at least one statistical analysis based on the validation results.
5. The device of claim 1 , wherein at least one validation case in the validation set is automatically derived by obtaining a sample data set, analyzing the sample data set, and identifying the at least one test condition from the analyzed sample data set.
6. The device of claim 1 , wherein at least one validation case in the validation set is derived by providing a user interface to enable manual entry of the at least one test condition.
7. The device of claim 1 , wherein at least one validation case in the validation set is obtained from a source, the source having previously derived the at least one test condition.
8. The device of claim 5 , wherein the computer executable instructions further cause the device to:
obtain the sample data set;
analyze the sample data set; and
automatically identify all test conditions to be validated for the at least one data set to validate.
9. The device of claim 8 , wherein the sample data set is analyzed by applying an automated process that uses a model derived using a machine learning process.
10. The device of claim 1 , wherein the computer executable instructions further cause the device to:
generate a notification indicative of at least one failure to trigger an investigation of the at least one data set.
11. The device of claim 1 , wherein validating the record comprises accessing a first record to be validated, incrementing through each of the at least one test condition to be validated for the first record and, for a second and any additional record to be validated, incrementing to a next record to increment through each of the at least one test condition.
12. The device of claim 11 , wherein the validation results comprise a pass or fail indication output as the validating increments through values in the records.
13. The device of claim 1 , wherein the data set is generated using financial data.
14. A method of validating data, the method executed by a device having a data interface coupled to a processor and comprising:
obtaining, via the data interface, at least one data set to be validated using a validation set determined according to at least one test condition, wherein the at least one dataset is an output of at least one statistical analysis on at least one input data set;
applying the validation set to the at least one data set to validate the data in the data set by, for at least one record in the at least one data set, validating the record according to the at least one test condition; and
outputting a validation result for the data set.
15. The method of claim 14 , further comprising:
in response to the validation result satisfying at least one criterion, transmitting the at least one data set to a process for consumption.
16. The method of claim 14 , wherein the validation set comprises at least one validation case, each validation case comprising at least one test condition.
17. The method of claim 14 , further comprising:
transmitting for adjusting, or adjusting, the at least one statistical analysis based on the validation results.
18. The method of claim 14 , wherein at least one validation case in the validation set is automatically derived by obtaining a sample data set, analyzing the sample data set, and identifying the at least one test condition from the analyzed sample data set.
19. The method of claim 14 , wherein at least one validation case in the validation set is derived by providing a user interface to enable manual entry of the at least one test condition.
20. A non-transitory computer readable medium for validating data, the computer readable medium comprising computer executable instructions for:
obtaining, via a data interface, at least one data set to be validated using a validation set determined according to at least one test condition, wherein the at least one dataset is an output of at least one statistical analysis on at least one input data set;
applying the validation set to the at least one data set to validate the data in the data set by, for at least one record in the at least one data set, validating the record according to the at least one test condition; and
outputting a validation result for the data set.