System and method for validating data
A system and method are provided for validating data. The method is executed by a device having a data interface coupled to a processor and includes obtaining a validation set comprising at least one validation case, each validation case comprising at least one test condition. The method also includes obtaining, via the data interface, at least one data set to be validated using the validation set. The method also includes applying the validation set to the at least one data set to validate the data in the data set by, for each record in the at least one data set, validating a value in the record according to the at least one test condition. The method also includes outputting a validation result for each record.
1. A device for validating data, the device comprising:
a processor;
a data interface coupled to the processor; and
a memory coupled to the processor, the memory storing computer executable instructions that when executed by the processor cause the processor to:
obtain a validation set comprising at least one validation case, each validation case comprising at least one test condition;
obtain, via the data interface, at least one data set to be validated using the validation set, wherein the at least one dataset is an output of at least one statistical analysis on at least one input data set;
apply the validation set to the at least one data set to validate the data in the data set by, for each record in the at least one data set, validating a value in the record according to the at least one test condition; and
output a validation result for each record.
2. The device of claim 1 , wherein at least one validation case is automatically derived by obtaining a sample data set, analyzing the sample data set, and identifying the at least one test condition from the analyzed sample data set.
3. The device of claim 1 , wherein at least one validation case is derived by providing a user interface to enable manual entry of the at least one test condition.
4. The device of claim 1 , wherein at least one validation case is obtained from a source, the source having previously derived the at least one test condition.
5. The device of claim 2 , wherein the computer executable instructions further cause the processor to:
obtain the sample data set;
analyze the sample data set; and
automatically identify all test conditions to be validated for the at least one data set to validate.
6. The device of claim 5 , wherein the sample data set is analyzed by applying an automated process that uses a model derived using a machine learning process.
7. The device of claim 1 , wherein the computer executable instructions further cause the processor to:
generate a notification indicative of at least one failure to trigger an investigation of the data set.
8. The device of claim 1 , wherein validating the value in the record comprises accessing a first record to be validated, incrementing through each of the at least one test condition to be validated for the first record and, for a second and any additional record to be validated, incrementing to a next record to increment through each of the at least one test condition.
9. The device of claim 8 , wherein the validation results comprise a pass or fail indication output as the validating increments through the values.
10. The device of claim 1 , wherein the data set is generated using financial data.
11. The device of claim 1 , wherein the computer executable instructions cause the processor to:
in response to the validation result satisfying at least one criterion, transmitting the at least one data set to a process for consumption.
12. A method of validating data, the method executed by a device having a data interface coupled to a processor and comprising:
obtaining a validation set comprising at least one validation case, each validation case comprising at least one test condition;
obtaining, via the data interface, at least one data set to be validated using the validation set, wherein the at least one dataset is an output of at least one statistical analysis on at least one input data set;
applying the validation set to the at least one data set to validate the data in the data set by, for each record in the at least one data set, validating a value in the record according to the at least one test condition; and
outputting a validation result for each record.
13. The method of claim 12 , wherein at least one validation case is automatically derived by obtaining a sample data set, analyzing the sample data set, and identifying the at least one test condition from the analyzed sample data set.
14. The method of claim 13 , further comprising:
obtain the sample data set;
analyze the sample data set; and
automatically identify all test conditions to be validated for the at least one data set to validate.
15. The method of claim 14 , wherein the sample data set is analyzed by applying an automated process that uses a model derived using a machine learning process.
16. The method of claim 12 , wherein validating the value in the record comprises accessing a first record to be validated, incrementing through each of the at least one test condition to be validated for the first record and, for a second and any additional record to be validated, incrementing to a next record to increment through each of the at least one test condition.
17. The method of claim 12 , wherein the computer executable instructions cause the processor to:
in response to the validation result satisfying at least one criterion, transmitting the at least one data set to a process for consumption.
18. A non-transitory computer readable medium for validating data, the computer readable medium comprising computer executable instructions for:
obtaining a validation set comprising at least one validation case, each validation case comprising at least one test condition;
obtaining, via a data interface, at least one data set to be validated using the validation set, wherein the at least one dataset is an output of at least one statistical analysis on at least one input data set;
applying the validation set to the at least one data set to validate the data in the data set by, for each record in the at least one data set, validating a value in the record according to the at least one test condition; and
outputting a validation result for each record.
19. The device of claim 1 , wherein the computer executable instructions cause the processor to:
transmit for adjusting, or adjust, the at least one statistical analysis based on the validation results.
20. The method of claim 12 , further comprising:
transmitting for adjusting, or adjusting, the at least one statistical analysis based on the validation results.