Data correctness and validation using validation definition language
Systems and methods are provided for generating extract-transform-load (“ETL”) machine learning (“ML”) pipeline validation rules based on user-input, wherein the ETL ML pipeline validation rules may be applicable to validate an ETL ML pipeline against multiple test datasets. The ETL ML pipeline validation rules may comprise compute-type validation rules for computing expected values of data structures within a dataset output by the ETL ML pipeline. The ETL ML pipeline validation rules may comprise check-type validation rules for checking whether data structures within a dataset output by the ETL ML pipeline have intended characteristics.
1. A method comprising:
receiving a validator rule description, wherein:
the validator rule description denotes a data field within an extract-transform-load (“ETL”) machine learning (“ML”) model as an intake data field;
the validator rule description denotes a data field within a referenced output schema as a result data field; and
based on the validator rule description, generating a validator rule comprising instructions; wherein when executed, the instructions cause a processor to:
compute a result data value based on an intake data value, wherein the intake data value is a value stored at a reappearance of the intake data field within a test dataset, and wherein a flag associated with the result data value is set, such that the result data value will not affect a later validation determination; and
write the result data value to a reappearance of the result data field within a received output schema.
2. The method of claim 1 , wherein when executing the instructions to compute the result data value, the instructions cause the processor to set a result data value to a specified value.
3. The method of claim 1 , wherein when executing the instructions to compute the result data value, the instructions cause the processor to set the result data value to the intake data value.
4. The method of claim 1 , wherein when executing the instructions to compute the result data value, the instructions cause the processor to set the result data value to a specified value if the intake data value is null.
5. The method of claim 1 , further wherein:
the validator rule description denotes a second data field within the ETL ML model as a second intake data field; and
when executing the instructions to compute the result data value, the instructions cause the processor to set the result data value to the intake data value added to a value stored at a reappearance of the second intake data field within the test dataset.
6. The method of claim 1 , further wherein:
the validator rule description denotes a second data field within the ETL ML model as a second intake data field; and
when executing the instructions to compute the result data value, the instructions cause the processor to set the result data value to the intake data value subtracted from a value stored at a reappearance of the second intake data field within the test dataset.
7. The method of claim 1 , further wherein:
the validator rule description denotes a second data field within the ETL ML model as a second intake data field; and
when executing the instructions to compute the result data value, the instructions cause the processor to set the result data value to the intake data value divided by a value stored at a reappearance of the second intake data field within the test dataset.
8. The method of claim 1 , further wherein:
the validator rule description denotes a second data field within the ETL ML model as a second intake data field; and
when executing the instructions to compute the result data value, the instructions cause the processor to set the result data value to the intake data value multiplied by a value stored at a reappearance of the second intake data field within the test dataset.
9. The method of claim 1 , wherein when executing the instructions to compute the result data value, the instructions cause the processor to set the result data value to a specified value if the intake data value satisfies a specified test.
10. The method of claim 1 , wherein when executing the instructions to compute the result data value, the instructions cause the processor to:
compute a converted intake data value, wherein the converted intake data value is the intake data value formatted according to a specified date format; and
set the result data value to the converted intake data value.
11. A method comprising:
receiving a validator rule description, wherein:
the validator rule description denotes a data field within an extract-transform-load (“ETL”) machine learning (“ML”) model as an intake data field;
the validator rule description denotes a data field within a referenced output schema as a result data field; and
based on the validator rule description, generating a validator rule comprising instructions; wherein when executed, the instructions cause a processor to:
compute a result data value based on an intake data value, wherein the intake data value is a value stored at a reappearance of the intake data field within a test dataset, and wherein the result data value is set to a specified value if the intake data value satisfies a specified test; and
write the result data value to a reappearance of the result data field within a received output schema.
12. The method of claim 11 , wherein the specified test is a logical comparison.
13. A method comprising:
receiving a validator rule description, wherein:
the validator rule description denotes a data field within an extract-transform-load (“ETL”) machine learning (“ML”) model as an intake data field;
the validator rule description denotes a data field within a referenced output schema as a result data field; and
based on the validator rule description, generating a validator rule comprising instructions; wherein when executed, the instructions cause a processor to:
compute a result data value based on an intake data value and a converted intake data value, wherein the intake data value is a value stored at a reappearance of the intake data field within a test dataset, and wherein the converted intake data value is the intake data value formatted according to a specified date format; and
set the result data value to the converted intake data value; and
write the result data value to a reappearance of the result data field within a received output schema.