Detecting data anomalies using artificial intelligence

1 . A computer-implemented method for improving data quality using an artificial intelligence (AI) model, the method comprising:

obtaining a dataset representing an observed value set for each variable in a variable set;

identifying, using a first AI model set, an anomaly set from the observed value set in the dataset by comparing an observed pattern set from the observed value set against one or more reference patterns that correspond to an expected value set for the variable set;

using a second AI model set to evaluate the identified anomaly set by:

generating an observed association rule set configured to cause generation of the observed value set in the dataset, and

comparing the observed association rule set with an expected association rule set to determine one or more observed association rules of the observed association rule set that correspond to the anomaly set;

determining a compliance indicator of the dataset based on a degree of satisfaction between the one or more observed association rules and the expected association rule set;

transmitting a representation of the compliance indicator of the dataset, wherein the representation indicates (a) the anomaly set and (b) a metadata set linked to the anomaly set;

responsive to obtaining a subsequent input, updating the representation by generating a computer-implemented command set configured to modify the one or more observed association rules corresponding to the anomaly set into alignment with one or more expected association rules.

2 . The computer-implemented method of claim 1 , wherein at least one model of the first AI model set or at least one model of the second AI model set are the same.

3 . The computer-implemented method of claim 1 , wherein updating the representation further comprises:

using a third AI model set to generate the computer-implemented command set.

4 . The computer-implemented method of claim 3 , further comprising:

executing the computer-implemented command set on the dataset.

5 . The computer-implemented method of claim 3 , wherein the third AI model set is configured to generate the computer-implemented command set by:

identifying a portion of the observed value set that corresponds to the one or more observed association rules that correspond to the anomaly set, and

associating the portion of the observed value set to one or more expected association rules of the expected association rule set, the one or more expected association rules being configured to adjust the portion of the observed value set to a corresponding expected value set.

6 . The computer-implemented method of claim 3 , further comprising:

selecting the third AI model set from multiple AI models using a respective set of performance metric values of each of the multiple AI models.

7 . The computer-implemented method of claim 1 , wherein the metadata set is associated with one or more of: (a) a data lineage of the anomaly set or (b) a version of the anomaly set.

8 . One or more non-transitory, computer-readable storage media comprising instructions thereon, wherein the instructions, when executed by at least one data processor of a system, cause the system to:

obtain a dataset representing an observed value set for each variable in a variable set;

identify, using a first artificial intelligence (AI) model set, an anomaly set from the observed value set in the dataset by comparing an observed pattern set from the observed value set against one or more reference patterns that correspond to an expected value set for the variable set;

use a second AI model set to evaluate the identified anomaly set by:

determining an observed rule set configured to cause generation of the observed value set in the dataset, and

determining one or more observed rules of the observed rule set that correspond to the anomaly set;

determine a compliance indicator of the dataset based on a degree of satisfaction between the one or more observed rules and an expected rule set;

transmit a representation of the compliance indicator of the dataset, wherein the representation indicates (a) the anomaly set and (b) a metadata set linked to the anomaly set; and

responsive to obtaining a subsequent input, update the representation by causing generation of a computer-implemented command set to remove one or more anomalies in the anomaly set from the observed value set in the dataset.

9 . The one or more non-transitory, computer-readable storage media of claim 8 , wherein the instructions further cause the system to:

cause execution of a set of workflows for a first type of anomaly, or

cause trigger of one or more alerts for a second type of anomaly.

10 . The one or more non-transitory, computer-readable storage media of claim 8 , wherein the representation indicates a compliance status of the dataset with a predefined guideline set associated with the expected rule set.

11 . The one or more non-transitory, computer-readable storage media of claim 8 , wherein the expected rule set is configured to dynamically adjust based on a time period associated with the observed rule set.

12 . The one or more non-transitory, computer-readable storage media of claim 8 , wherein the dataset includes one or more of: text documents, emails, chat logs, images, or voice recordings.

13 . The one or more non-transitory, computer-readable storage media of claim 8 , wherein the representation is displayed on a user interface of a computing device.

14 . The one or more non-transitory, computer-readable storage media of claim 8 , wherein at least one model of the first AI model set or at least one model of the second AI model set are the same.

15 . A system comprising:

at least one hardware processor; and

at least one non-transitory memory storing instructions, which, when executed by the at least one hardware processor, cause the system to:

obtain a dataset representing an observed value set for each variable in a variable set;

identify, using a first artificial intelligence (AI) model set, an anomaly set from the observed value set in the dataset by comparing an observed pattern set from the observed value set against one or more reference patterns that correspond to an expected value set for the variable set;

use a second AI model set to evaluate the identified anomaly set by:

determining an observed rule set configured to cause generation of the observed value set in the dataset, and

determining one or more observed rules of the observed rule set that correspond to the anomaly set;

determine a compliance indicator of the dataset based on a degree of satisfaction between the one or more observed rules and an expected rule set;

transmit a representation of the compliance indicator of the dataset, wherein the representation indicates one or more of: (a) the anomaly set or (b) a metadata set linked to the anomaly set; and

responsive to obtaining a subsequent input, update the representation by causing execution of a computer-implemented command set to remove one or more anomalies in the anomaly set from the observed value set in the dataset.

16 . The system of claim 15 , wherein determine the compliance indicator further comprises:

comparing the degree of satisfaction against a predefined threshold, wherein the threshold is determined via a user input.

17 . The system of claim 15 , wherein the system is further caused to:

generate the computer-implemented command set configured to modify the one or more observed rules corresponding to the anomaly set into alignment with one or more expected rules.

18 . The system of claim 17 , wherein the system is further caused to:

execute the computer-implemented command set on the dataset.

19 . The system of claim 15 , wherein the metadata set is associated with one or more of: (a) a data lineage of the anomaly set or (b) a version of the anomaly set.

20 . The system of claim 15 , wherein at least one model of the first AI model set or at least one model of the second AI model set are the same.