System and method for multivariate anomaly detection
Disclosed are systems, methods, and devices for data anomaly detection. A signal reflective of an input data set having a plurality of dimensions is received. Co-variance across said plurality of dimensions is assessed. Upon said assessing, at least a portion of the input data set is transformed into a dimensionality-reduced data set. For each given data point in the dimensionality-reduced data set, an anomaly score informative of whether said given data point is an anomaly is calculated.
1 . A computer-implemented method for data anomaly detection, said method comprising:
receiving a signal reflective of an input data set having a plurality of dimensions, the input data set associated with an audit process of an incident ticket management system;
processing the input data set to determine at least one attribute of each variable in the input data set, the at least one attribute including whether a variable is continuous or discrete;
providing the input data set and the at least one attribute of each variable to a dimensionality reduction architecture comprising a first dimensionality reducer and a second dimensionality reducer; wherein the first dimensionality reducer is configured to execute a first co-variance test, and the second dimensionality reducer is configured to execute a second co-variance test; wherein the first co-variance test and/or the second co-variance test includes determining with at least one processor co-variance across said plurality of dimensions, wherein said assessing co-variance includes, for each pair of two variables in a plurality of pairwise combinations of variables in the input data set;
transforming, with the dimensionality reduction architecture, the input data set into a dimensionality-reduced data set using the first dimensionality reducer or the second dimensionality reducer, wherein the dimensionality reduction architecture selects the first dimensionality reducer or the second dimensionality reducer based on a count of the number of continuous variables in the input data set;
wherein the transforming includes: when the selected first or second dimensionality reducer determines that the two variables associated with their respective co-variance test is statistically significant based on a pre-defined threshold, transforming at least a portion of said input data set into a dimensionality-reduced data set by removing data for one of the two variables associated with the respective co-variance test;
for each given data point in said dimensionality-reduced data set, calculating an anomaly score informative of whether said given data point is an anomaly, wherein for at least one data point in the dimensionality-reduced data set, calculating the anomaly score for the at least one data point comprises calculating a quantity of an interquartile range (IQR) based on a distance between the at least one data point and a median value, wherein an upper bound of the anomaly score is set to twice the IQR;
generating signals representing a graphical indicator for rendering within a graphical user interface to indicate the anomaly score, wherein the generated signals comprise a signal for displaying the graphical indicator in a shading determined based on a respective attribute of the at least one data point, wherein the graphical indicator represents an identifier of an application as a cause for the anomaly associated with the incident ticket management system; and
transmitting the signals representing the graphical indicator to a client device for display.
2 . The computer-implemented method of claim 1 , further comprising calculating an aggregate anomaly score by combining a plurality of said anomaly scores.
3 . The computer-implemented method of claim 1 , wherein said at least one attribute includes whether said variable is normally distributed.
4 . The computer-implemented method of claim 1 , wherein said at least one attribute includes whether said variable is continuous.
5 . The computer-implemented method of claim 1 , wherein said plurality of co-variance tests includes at least two of a Pearson's correlation, a Spearman's correlation, an F-Test, a T-Test, a Kruskal-Wallis, a Mann-Whitney U Test, and a χ 2 Test.
6 . The computer-implemented method of claim 1 , wherein at least one of said assessing co-variance and said transforming said input data set includes applying steps of principle component analysis.
7 . The computer-implemented method of claim 6 , wherein said applying is upon determining that a count of continuous variables in said input data set exceeds a pre-defined threshold.
8 . The computer-implemented method of claim 1 , further comprising identifying a categorical variable in said input data set.
9 . The computer-implemented method of claim 8 , wherein said calculating said anomaly score for said categorical variable comprises calculating an inverse frequency of a class of said given data point.
10 . The computer-implemented method of claim 1 , further comprising dividing said input data set into a plurality of subsets.
11 . The computer-implemented method of claim 10 , wherein said at least a portion of said input data set is one of said plurality of subsets.
12 . The computer-implemented method of claim 10 , wherein said transforming said at least a portion of said input data set includes separately transforming each of said plurality of subsets.
13 . A computer-implemented system for data anomaly detection, the system comprising:
at least one processor;
memory in communication with the at least one processor, and software code stored in the memory, which when executed by the at least one processor causes the system to:
receive a signal reflective of an input data set having a plurality of dimensions, the input data set associated with an audit process of an incident ticket management system;
process the input data set to determine at least one attribute of each variable in the input data set, the at least one attribute including whether a variable is continuous or discrete;
provide the input data set and the at least one attribute of each variable to a dimensionality reduction architecture comprising a first dimensionality reducer and a second dimensionality reducer; wherein the first dimensionality reducer executes a first co-variance test, and the second dimensionality reducer executes a second co-variance test; wherein the first co-variance test and/or the second co-variance test includes determining with the at least one processor co-variance across said plurality of dimensions, wherein said assessing co-variance includes, for each pair of two variables in a plurality of pairwise combinations of variables in the input data set;
transform, with the dimensionality reduction architecture, the input data set into a dimensionality-reduced data set using the first dimensionality reducer or the second dimensionality reducer, wherein the dimensionality reduction architecture selects the first dimensionality reducer or the second dimensionality reducer based on a count of the number of continuous variables in the input data set;
wherein the transforming includes: when the selected first or second dimensionality reducer determines that the two variables associated with the respective co-variance test is statistically significant based on a pre-defined threshold, transform at least a portion of said input data set into a dimensionality-reduced data set by removing data for one of the two variables associated with the respective co-variance test;
for each given data point in said dimensionality-reduced data set, calculate an anomaly score informative of whether said given data point is an anomaly;
wherein for at least one data point in the dimensionality-reduced data set, calculating the anomaly score for the at least one data point comprises calculating a quantity of an interquartile range (IQR) based on a distance between the at least one data point and a median value, wherein an upper bound of the anomaly score is set to twice the IQR;
generate signals representing a graphical indicator for rendering within a graphical user interface to indicate the anomaly score, wherein the generated signals comprise a signal for displaying the graphical indicator in a shading determined based on a respective attribute of the at least one data point, wherein the graphical indicator represents an identifier of an application as a cause for the anomaly associated with the incident ticket management system; and
transmit the signals representing the graphical indicator to a client device for display.
14 . A non-transitory computer-readable medium or media having stored thereon machine interpretable instructions which, when executed by a processor, cause the processor to perform a computer implemented method of data anomaly detection, the method comprising:
receiving a signal reflective of an input data set having a plurality of dimensions, the input data set associated with an audit process of an incident ticket management system;
processing the input data set to determine at least one attribute of each variable in the input data set, the at least one attribute including whether a variable is continuous or discrete;
providing the input data set and the at least one attribute of each variable to a dimensionality reduction architecture comprising a first dimensionality reducer and a second dimensionality reducer; wherein the first dimensionality reducer is configured to execute a first co-variance test, and the second dimensionality reducer is configured to execute a second co-variance test; wherein the first co-variance test and/or the second co-variance test includes determining with the processor co-variance across said plurality of dimensions, wherein said assessing co-variance includes, for each pair of two variables in a plurality of pairwise combinations of variables in the input data set;
transforming, with the dimensionality reduction architecture, the input data set into a dimensionality-reduced data set using the first dimensionality reducer or the second dimensionality reducer, wherein the dimensionality reduction architecture selects the first dimensionality reducer or the second dimensionality reducer based on a count of the number of continuous variables in the input data set;
wherein the transforming includes: when the selected first or second dimensionality reducer determines that the two variables associated with the respective co-variance test is statistically significant based on a pre-defined threshold, transforming at least a portion of said input data set into a dimensionality-reduced data set by removing data for one of the two variables associated with the respective co-variance test;
for each given data point in said dimensionality-reduced data set, calculating an anomaly score informative of whether said given data point is an anomaly;
wherein for at least one data point in the dimensionality-reduced data set, calculating the anomaly score for the at least one data point comprises calculating a quantity of an interquartile range (IQR) based on a distance between the at least one data point and a median value, wherein an upper bound of the anomaly score is set to twice the IQR;
generating signals representing a graphical indicator for rendering within a graphical user interface to indicate the anomaly score, wherein the generated signals comprise a signal for displaying the graphical indicator in a shading determined based on a respective attribute of the at least one data point, wherein the graphical indicator represents an identifier of an application as a cause for the anomaly associated with the incident ticket management system; and
transmitting the signals representing the graphical indicator to a client device for display.
15 . The system of claim 13 , wherein the software code, when executed by the at least one processor causes the system to: calculate an aggregate anomaly score by combining a plurality of said anomaly scores.
16 . The system of claim 13 , wherein said at least one attribute includes whether said variable is normally distributed.
17 . The system of claim 13 , wherein said at least one attribute includes whether said variable is continuous.
18 . The system of claim 13 , wherein said plurality of co-variance tests includes at least two of a Pearson's correlation, a Spearman's correlation, an F-Test, a T-Test, a Kruskal-Wallis, a Mann-Whitney U Test, and a χ 2 Test.