Systems and methods for detecting anomalies in data records across databases with different data schemas
Embodiments described herein provide a probabilistic approach using a probabilistic classifier to detect anomalous categorical data, e.g., data points in non-numerical data fields. Specifically, the probabilistic classifier may generate a conditional probability distribution of a data attribute, such as the “country of origin” field in a bond data record, conditioned on various other data attributes of the bond data record obtained from different databases. The generated conditional probability corresponding to the data attribute of the store non-numerical value, e.g., the “country of origin” equals a certain country code as stored, may indicate whether a data anomaly exists with the stored value for this particular data attribute. When an anomaly is detected with a data attribute, the categorical value associated with the highest conditional probability corresponding to the data attribute may be provided as the suggested value for the data attribute.
1 . A method for anomaly detection of categorical data, the method comprising:
retrieving, from a first database storing a first plurality of data records compliant with a first data schema, a first data record having at least a first data attribute having an actual value that takes a format compliant with the first data schema;
determining, by a processor, from a second database storing a second plurality data records compliant with a second data schema different from the first data schema, a second data record as related to the first data attribute;
determining, from a third database storing a third plurality data records compliant with a third data schema different from the first data schema or the second data schema, a third data record as related to the first data attribute;
generating, by the processor, an input set of tokens formed by data attributes of the second data record that are compliant with the second data schema;
appending data attributes from the third data record as tokens to the input set of tokens irrespective of a difference among the first, the second and the third data schemas;
passing the input set of tokens to a neural network based classification model that is pre-trained with a set of classes corresponding to a set of candidate values for the first data attribute;
generating, from the neural network based classification model, a conditional distribution of a predicted value of the first data attribute that belongs to the first data record from the first database, using an input of the input set of tokens comprising the data attributes in the second data record that is from the second database and the data attributes from the third data record that is from the third database, wherein the neural network based classification model is trained by:
obtaining training data of at least a first set of data records compliant with the first data schema and at least a second set of data records compliant with the second data schema, wherein at least one data record of the first set of data records contains the first data attribute;
generating, by the neural network based classification model, a training output probability distribution for the first data attribute of the at least one data record that belongs to the first set of data records based on a training input comprising at least a set of data attributes relating to the first data attribute but belonging to the second set of data records; and
training the neural network based classification model based on a loss metric comparing the training output probability distribution for the first data attribute and an actual value of the first data attribute from the at least one data record of the first set data records;
determining whether the first data attribute is an anomaly based on the actual value compliant with the first data schema and the conditional distribution of the predicted value of the first data attribute among the set of candidate values;
determining, from the second database, a fourth data record as related to a second data attribute of the first data record; and
generating, by the trained neural network based classification model, a joint conditional probability distribution for the first data attribute and the second data attribute, wherein the joint conditional probability distribution indicates whether a combination of actual values of the first data attribute and the second data attribute in the first data record is an anomaly.
2 . The method of claim 1 , wherein the first data attribute corresponds to the set of candidate values, and each candidate value from the set of candidate values is compliant with the first data schema other than a numerical value.
3 . The method of claim 1 , wherein determining the second data record as related to the first data attribute comprises:
extracting one or more key terms corresponding to the first data attribute and the first data record; and
forming a query on the second database based on the one or more key terms for the second data record.
4 . The method of claim 1 , further comprising:
incorporating a second data attribute from the first data record into the input set of tokens.
5 . The method of claim 4 , wherein the data attributes of the second data record that are compliant with the second data schema and the second data attribute that is compliant with the first data schema are each read by the processor as a token irrespective of a difference between the first data schema and the second data schema.
6 . The method of claim 1 , wherein
training the neural network based classification model further comprises
updating at least one or more parameters of the neural network based classification model based on the loss metric.
7 . The method of claim 1 , wherein determining whether the first data attribute is an anomaly based on an actual value corresponding to the first data attribute in the first data schema and the conditional distribution of the first data attribute among the set of candidate values further comprises:
determining whether a respective conditional probability corresponding to the actual value of the first data attribute is greater than a threshold; and
generating an error message when the respective conditional probability is lower than the threshold.
8 . The method of claim 7 , further comprising:
in response to determining that the respective conditional probability corresponding to the actual value of the first data attribute is lower than the threshold, determining a suggested value corresponding to a highest probability among the conditional distribution of the first data attribute.
9 . A system for anomaly detection of categorical data, the system comprising:
a memory storing a classification model;
a communication interface configured to retrieve, from a first database storing a first plurality of data records compliant with a first data schema, a first data record having at least a first data attribute having an actual value that takes a format compliant with the first data schema; and
a processor configured to:
determine, from a second database storing a second plurality data records compliant with a second data schema different from the first data schema, a second data record as related to the first data attribute;
determine, from a third database storing a third plurality data records compliant with a third data schema different from the first data schema or the second data schema, a third data record as related to the first data attribute;
generate an input set of tokens based on data attributes of the second data record that are compliant with the second data schema;
append data attributes from the third data record as tokens to the input set of tokens irrespective of a difference among the first, the second and the third data schemas;
pass the input set of tokens to a neural network based classification model that is pre-trained with a set of classes corresponding to a set of candidate values for the first data attribute;
generate, from the neural network based classification model, a conditional distribution of a predicted value of the first data attribute that belongs to the first data record from the first database, using an input of the input set of tokens comprising the data attributes in the second data record that is from the second database and the data attributes from the third data record that is from the third database, wherein the neural network based classification model is trained by:
obtaining training data of at least a first set of data records compliant with the first data schema and at least a second set of data records compliant with the second data schema, wherein at least one data record of the first set of data records contains the first data attribute;
generating, by the neural network based classification model, a training output probability distribution for the first data attribute based on a training input comprising at least a set of data attributes from the second set of data records; and
training the neural network based classification model based on a loss metric comparing the training output probability distribution and a value of the first data attribute from the at least one data record of the first set data records;
determine whether the first data attribute is an anomaly based on the actual value compliant with the first data schema and the conditional distribution of the predicted value of the first data attribute among the set of candidate values;
determine, from the second database, a fourth data record as related to a second data attribute of the first data record; and
generate, by the trained neural network based classification model, a joint conditional probability distribution for the first data attribute and the second data attribute, wherein the joint conditional probability distribution indicates whether a combination of actual values of the first data attribute and the second data attribute in the first data record is an anomaly.
10 . The system of claim 9 , wherein the first data attribute corresponds to the set of candidate values, and each candidate value from the set of candidate values is compliant with the first data schema other than a numerical value.
11 . The system of claim 9 , wherein the processor is further configured to determine the second data record as related to the first data attribute by:
extracting one or more key terms corresponding to the first data attribute and the first data record; and
forming a query on the second database based on the one or more key terms for the second data record.
12 . The system of claim 11 , wherein the processor is further configured to:
incorporate a second data attribute from the first data record into the input set of tokens.
13 . The system of claim 12 , wherein the data attributes of the second data record that are compliant with the second data schema and the second data attribute that is compliant with the first data schema are each read by the processor as a token irrespective of a difference between the first data schema and the second data schema.
14 . The system of claim 9 , wherein the processor is further configured to determine whether the first data attribute is an anomaly based on an actual value corresponding to the first data attribute in the first data schema and the conditional distribution of the first data attribute among the set of candidate values by:
determining whether a respective conditional probability corresponding to the actual value of the first data attribute is greater than a threshold; and
generating an error message when the respective conditional probability is lower than the threshold.
15 . The system of claim 14 , wherein the processor is further configured to determine a suggested value corresponding to a highest probability among the conditional distribution of the first data attribute in response to determining that the respective conditional probability corresponding to the actual value of the first data attribute is lower than the threshold.
16 . The system of claim 14 , wherein the error message includes a decision tree showing a link between the first data attribute to sequentially connected data attributes from other databases down a tree structure.
17 . The system of claim 16 , wherein the decision tree shows a truth score computed by the classification model for each root to leaf path indicating a conditional likelihood of the first data attribute conditioned on the sequentially connected data attributes along the respective root to leaf path.
18 . A non-transitory processor-readable medium storing processor-executable instructions for anomaly detection of categorical data, the instructions being executable by a processor to perform operations comprising:
retrieving, from a first database storing a first plurality of data records compliant with a first data schema, a first data record having at least a first data attribute having an actual value that takes a format compliant with the first data schema;
determining, by a processor, from a second database storing a second plurality data records compliant with a second data schema different from the first data schema, a second data record as related to the first data attribute;
determining, from a third database storing a third plurality data records compliant with a third data schema different from the first data schema or the second data schema, a third data record as related to the first data attribute;
generating, by the processor, an input set of tokens formed by data attributes of the second data record that are compliant with the second data schema;
appending data attributes from the third data record as tokens to the input set of tokens irrespective of a difference among the first, the second and the third data schemas;
passing the input set of tokens to a neural network based classification model that is pre-trained with a set of classes corresponding to a set of candidate values for the first data attribute;
generating, from the neural network based classification model, a conditional distribution of a predicted value of the first data attribute that belongs to the first data record from the first database, using an input of the input set of tokens comprising the data attributes in the second data record that is from the second database and the data attributes from the third data record that is from the third database, wherein the neural network based classification model is trained by:
obtaining training data of at least a first set of data records compliant with the first data schema and at least a second set of data records compliant with the second data schema, wherein at least one data record of the first set of data records contains the first data attribute;
generating, by the neural network based classification model, a training output probability distribution for the first data attribute based on a training input comprising at least a set of data attributes from the second set of data records; and
training the neural network based classification model based on a loss metric comparing the training output probability distribution and a value of the first data attribute from the at least one data record of the first set data records;
determining whether the first data attribute is an anomaly based on the actual value compliant with the first data schema and the conditional distribution of the predicted value of the first data attribute among the set of candidate values;
determining, from the second database, a fourth data record as related to a second data attribute of the first data record; and
generating, by the trained neural network based classification model, a joint conditional probability distribution for the first data attribute and the second data attribute, wherein the joint conditional probability distribution indicates whether a combination of actual values of the first data attribute and the second data attribute in the first data record is an anomaly.
19 . A method for anomaly detection of categorical data, the method comprising:
retrieving, from a first financial database storing a first plurality of financial instrument records compliant with a first financial data schema, a first financial instrument record having at least a first non-numerical data attribute compliant with the first financial data schema;
receiving a request to verify a data accuracy of the first non-numerical data attribute in the first financial instrument record;
obtaining, from at least a second financial database storing a second plurality of financial instrument records compliant with a second financial data schema different from the first financial data schema, a second financial data record relating to a same financial instrument with the first financial instrument record,
wherein the second financial data record has a second non-numerical data attribute compliant with the second financial data scheme;
obtaining, from a third financial database storing a third plurality of financial instrument records compliant with a third financial data schema different from the first financial data schema and the second financial data scheme, a third financial data record relating to the same financial instrument with the first financial instrument record,
wherein the third financial data record has a third non-numerical data attribute compliant with the third financial data scheme;
generating, by the processor, an input set of tokens based on the first non-numerical data attribute and the second non-numerical data attribute irrespective of their respective different data schema;
append data attributes from the third data record as tokens to the input set of tokens irrespective of a difference among the first, the second and the third data schemas;
passing the input set of tokens to a neural network based classification model that is pre-trained, with datasets from both the first financial database and the second financial database, for classifying an input into a set of classes corresponding to a set of candidate values for the first non-numeric data attribute, wherein the neural network based classification model is trained by:
obtaining training data of at least a first set of data records compliant with the first data schema and at least a second set of data records compliant with the second data schema, wherein at least one data record of the first set of data records contains the first data attribute;
generating, by the neural network based classification model, a training output probability distribution for the first data attribute based on a training input comprising at least a set of data attributes from the second set of data records; and
training the neural network based classification model based on a loss metric comparing the training output probability distribution and a value of the first data attribute from the at least one data record of the first set data records;
generating, from the neural network based classification model, a conditional distribution of a predicted value of the first non-numeric data attribute that belongs to the first financial record from the first financial database using an input of the input set of tokens comprising at least the second non-numeric data attribute in the second financial data record from the second database and the data attributes from the third data record that is from the third database;
determining whether the first non-numeric data attribute is an anomaly based on the conditional distribution of the predicted value of the first financial data attribute among the set of candidate values;
determining, from the second database, a fourth financial data record as related to a second non-numeric data attribute of the first financial data record; and
generating, by the trained neural network based classification model, a joint conditional probability distribution for the first non-numeric data attribute and the second non-numeric data attribute, wherein the joint conditional probability distribution indicates whether a combination of actual values of the first non-numeric data attribute and the second non-numeric data attribute in the first financial data record is an anomaly; and
generating an output message including the conditional distribution of the first non-numeric data attribute or the joint conditional probability distribution.