Systems and methods for dynamically generating new data rules
Systems, apparatuses, methods, and computer program products are disclosed for dynamically generating a new data rule. An example method includes receiving a data rule including an operation to be applied to a data entry and determining regarding whether the data rule belongs to a set of previously-reviewed data rules. The example method further includes, when the data rule does not belong to the set of previously-reviewed data rules, providing the data rule to a user for human validation and receiving human validation. The example method further includes, causing an update, by rule generation circuitry, of an element of a machine learning model, where the element corresponds to the data rule where the update changes a weight value based on the validation and generating the new data rule based on the updated machine learning model. The example method further includes replacing the data rule with the new data rule.
1. A method for dynamically generating a new data rule, the method comprising:
receiving, by data transform circuitry, a data rule, wherein the data rule comprises an operation to be applied to a data entry, wherein the data rule is generated using a machine learning model, wherein the data rule corresponds to an element of the machine learning model, wherein the data rule belongs to an ETL process from a set of ETL processes;
making a determination, by rule validation circuitry, regarding whether the data rule belongs to a set of previously-reviewed data rules;
in an instance in which the determination indicates that the data rule does not belong to the set of previously-reviewed data rules, providing, by the rule validation circuitry, the data rule to a user for human validation;
receiving, by the rule validation circuitry, an indication of human validation of the data rule from the user, wherein the human validation indicates one item from a list comprising “valid,” “invalid,” and “ambiguous,” wherein “ambiguous” indicates allowing automated updating of the data rule;
performing, by rule generation circuitry and based on the indication of human validation, an optimization operation using the new data rule, wherein performing the optimization comprises, in an instance in which the human validation indicates “ambiguous”:
causing an update, by rule generation circuitry, of the element of the machine learning model corresponding to the data rule, wherein the machine learning model comprises a weight value, wherein the update changes the weight value based on the indication of human validation, wherein causing an update of the element of the machine learning model results in an updated machine learning model;
modifying, by the rule generation circuitry, the machine learning model based on the set of ETL processes and a set of quality scores associated with the set of ETL processes;
generating, by the rule generation circuitry, the new data rule based on the updated machine learning model;
determining, by the rule generation circuitry, whether the new data rule passes a confidence threshold; and
in an instance in which the new data rule passes the confidence threshold, replacing, by the data transform circuitry and based on determining that the new data rule passes the confidence threshold, the data rule with the new data rule.
2. The method of claim 1 , wherein the rule validation circuitry comprises a field programmable gate array.
3. The method of claim 1 , wherein the data rule belongs to an extract-transform-load (ETL) process, the method further comprising:
training, by the rule generation circuitry, a machine learning model using existing data rules from existing ETL processes.
4. The method of claim 1 , wherein performing the optimization operation using the new data rule further comprises, in an instance in which the human validation indicates “ambiguous”:
in an instance in which the new data rule does not pass the confidence threshold, repeating generation of the new data rule.
5. The method of claim 1 , wherein causing the update of the element of the machine learning model comprises:
receiving, by the rule generation circuitry, the set of ETL processes; and
assigning, by the rule generation circuitry, a quality score from the set of quality scores to each ETL process from the set of ETL processes.
6. The method of claim 1 , wherein the data rule belongs to an ETL process, wherein performing the optimization operation using the new data rule comprises, in an instance in which the human validation identifies the data rule is valid, adding, by the data transform circuitry, the data rule to the ETL process.
7. The method of claim 1 , wherein performing the optimization operation using the new data rule comprises, in an instance in which the human validation identifies the data rule as invalid, discarding, by the data transform circuitry, the data rule.
8. The method of claim 1 ,
wherein the set of previously-reviewed data rules is continually updated upon reviewing a subsequent data rule,
wherein a set of previously reviewed data rule validation decisions is stored and continually updated upon receiving a subsequent indication of human validation.
9. The method of claim 8 , further comprising, in an instance in which the data rule belongs to the set of previously reviewed data rules:
locating, by the rule validation circuitry, a validation decision in the set of previously reviewed data rule validation decisions, wherein the validation decision is related to the data rule; and
applying, by the rule validation circuitry, the validation decision to the data rule.
10. An apparatus for dynamically generating a new data rule, the apparatus comprising:
data transform circuitry configured to receive a data rule, wherein the data rule comprises an operation to be applied to a data entry, wherein the data rule is generated using a machine learning model, wherein the data rule corresponds to an element of the machine learning model, wherein the data rule belongs to an ETL process from a set of ETL processes;
rule validation circuitry configured to:
make a determination regarding whether the data rule belongs to a set of previously-reviewed data rules,
in an instance in which the determination indicates that the data rule does not belong to the set of previously-reviewed data rules, provide the data rule to a user for human validation, and
receive an indication of human validation of the data rule from the user wherein the human validation indicates one item from a list comprising “valid,” “invalid,” and “ambiguous,” wherein “ambiguous” indicates allowing automated updating of the data rule; and
rule generation circuitry configured to perform an optimization operation based on the indication of human validation and using the new data rule,
wherein the rule generation circuitry is configured such that performing the optimization operation further comprises, in an instance in which the human validation indicates “ambiguous”:
causing an update of an element of a machine learning model, wherein the element corresponds to the data rule, wherein the update changes a weight value based on the indication of human validation, wherein causing an update of the element of the machine learning model results in an updated machine learning model;
modifying the machine learning model based on the set of ETL processes and a set of quality scores associated with the set of ETL processes;
generating the new data rule based on the updated machine learning model;
determining whether the new data rule passes a confidence threshold; and
in an instance in which the new data rule passes the confidence threshold, replacing, based on determining that the new data rule passes the confidence threshold, the data rule with the new data rule.
11. The apparatus of claim 10 , wherein the rule validation circuitry comprises a field programmable gate array.
12. The apparatus of claim 10 , wherein the data rule belongs to an extract-transform-load (ETL) process.
13. The apparatus of claim 12 , wherein the rule generation circuitry is further configured to train a machine learning model using existing data rules from existing ETL processes.
14. The apparatus of claim 10 , wherein the rule generation circuitry is further configured so that causing the update of the element of the machine learning model comprises:
receiving the set of ETL processes; and
assigning a quality score from the set of quality scores to each ETL process from the set of ETL processes.
15. The apparatus of claim 10 , wherein the data rule belongs to an ETL process, wherein the data transform circuitry is configured so that performing the optimization operation using the new data rule comprises, in an instance in which the human validation identifies the data rule is valid, adding the data rule to the ETL process.
16. The apparatus of claim 10 , wherein the data transform circuitry is configured so that performing the optimization operation using the new data rule comprises, in an instance in which the human validation identifies the data rule as invalid, discarding the data rule.
17. A computer program product for dynamically generating a new data rule, the computer program product comprising at least one non-transitory computer-readable storage medium storing software instructions that, when executed, cause an apparatus to:
receive a data rule, wherein the data rule comprises an operation to be applied to a data entry, wherein the data rule belongs to an ETL process from a set of ETL processes;
make a determination regarding whether the data rule belongs to a set of previously-reviewed data rules;
in an instance in which the determination indicates that the data rule does not belong to the set of previously-reviewed data rules, provide the data rule to a user for human validation;
receive an indication of human validation of the data rule from the user, wherein the human validation indicates one item from a list comprising “valid,” “invalid,” and “ambiguous,” wherein “ambiguous” indicates allowing automated updating of the data rule; and
in an instance in which the human validation indicates “ambiguous”:
cause an update of an element of a machine learning model, wherein the element corresponds to the data rule, wherein the update changes a weight value based on the indication of human validation, wherein causing an update of the element of the machine learning model results in an updated machine learning model,
modify the machine learning model based on the set of ETL processes and a set of quality scores associated with the set of ETL processes,
generate the new data rule based on the updated machine learning model,
determine whether the new data rule passes a confidence threshold, and
in an instance in which the new data rule passes the confidence threshold, replace, based on determining that the new data rule passes the confidence threshold, the data rule with the new data rule.
18. The computer program product of claim 17 , wherein the data rule belongs to an extract-transform-load (ETL) process.
19. The computer program product of claim 18 , wherein the software instructions further cause the apparatus to:
train the machine learning model using existing data rules from existing ETL processes.