CONVOLUTIONAL NEURAL NETWORK (CNN)-BASED ANOMALY DETECTION
The technology disclosed determines which field values in a set of unique field values for a particular field in a fielded dataset are anomalous using six similarity measures. A factor vector is generated per similarity measure and combined to form an input matrix. A convolutional neural network processes the input matrix to generate evaluation vectors. A fully-connected network evaluates the evaluation vectors to generate an anomaly scalar for a particular unique field value. Thresholding is applied to anomaly scalar to determine whether the particular unique field value is anomalous.
1 . A system of an anomaly detection network, including:
a communication interface receiving a set of input values in a format of an input matrix;
a memory storing the anomaly detection network comprising one or more convolution layers and a fully-connected layer connected to the one or more convolution layers;
one or more processors coupled to the memory to perform operations based on the anomaly detection network, comprising:
generating, by the one or more convolution layers, one or more evaluation vectors from the input matrix,
generating, by the fully-connected layer, accumulated element-wise weighted sums of the one or more evaluation vectors to form an output vector, and
determining an indication that suggests an anomaly in the set of input values based on the output vector.
2 . The system of claim 1 , wherein the set of input values are for a particular field in a fielded dataset.
3 . The system of claim 1 , wherein the one or more convolution layers includes at least one convolutional filter that convolve a first row of the input matrix to compute a first entry in a first evaluation vector in the one or more evaluation vectors.
4 . The system of claim 1 , wherein the memory further stores a factor vector calculator connected to the one or more convolution layers, and the factor vector calculator comprises a plurality of similarity measure calculators configured to apply a plurality of similarity measures to the set of input values, respectively, and
wherein the factor vector calculator is further configured to compute factor vectors based on the plurality of similarity measures to form the input matrix.
5 . The system of claim 3 , wherein the plurality of similarity measures include any combination of semantic similarity, syntactic similarity, soundex similarity, character-by-character format similarity, field length similarity, and dataset frequency similarity.
6 . The system of claim 1 , wherein the memory further stores a non-linear module connected to the fully-connected layer,
wherein the non-linear module is configured to normalize the output vector by any of a sigmoid function, a hyperbolic tangent (tanh) function, a rectified linear unit (ReLU) and a leaky ReLU.
7 . The system of claim 1 , wherein the operations further comprise:
determining whether the set of input values contains the anomaly by comparing each entry in the output vector with a pre-defined threshold.
8 . The system of claim 1 , wherein the operations further comprise:
determining one or more suggestion candidates to replace a particular input value in the set of input values based on a corresponding entry that corresponds to the particular input value in the output vector.
9 . A method for anomaly detection in a set of fields values, the method comprising:
receiving, via a communication interface, a set of input values in a format of an input matrix;
generating, by one or more convolution layers, one or more evaluation vectors from the input matrix,
generating, by a fully-connected layer connected to the one or more convolution layers, accumulated element-wise weighted sums of the one or more evaluation vectors to form an output vector, and
determining an indication that suggests an anomaly in the set of input values based on the output vector.
10 . The method of claim 9 , wherein the one or more convolution layers includes at least one convolutional filter that convolve a first row of the input matrix to compute a first entry in a first evaluation vector in the one or more evaluation vectors.
11 . The method of claim 9 , further comprising:
applying, by a factor vector calculator connected to the one or more convolution layers, a plurality of similarity measures to the set of input values, respectively; and
computing factor vectors based on the plurality of similarity measures to form the input matrix.
12 . The method of claim 11 , wherein the plurality of similarity measures include any combination of semantic similarity, syntactic similarity, soundex similarity, character-by-character format similarity, field length similarity, and dataset frequency similarity.
13 . The method of claim 9 , further comprising:
normalizing, by a non-linear function module, the output vector by any of a sigmoid function, a hyperbolic tangent (tanh) function, a rectified linear unit (ReLU) and a leaky ReLU.
14 . The method of claim 9 , further comprising:
determining whether the set of input values contains the anomaly by comparing each entry in the output vector with a pre-defined threshold.
15 . The method of claim 1 , further comprising:
determining one or more suggestion candidates to replace a particular input value in the set of input values based on a corresponding entry that corresponds to the particular input value in the output vector.
16 . A non-transitory processor-executable storage medium storing a plurality of processor-executable instructions for anomaly detection in a set of fields values, the instructions being executed by a processor to perform operations comprising:
receiving a set of input values in a format of an input matrix;
generating, by one or more convolution layers, one or more evaluation vectors from the input matrix,
generating, by a fully-connected layer connected to the one or more convolution layers, accumulated element-wise weighted sums of the one or more evaluation vectors to form an output vector, and
determining an indication that suggests an anomaly in the set of input values based on the output vector.
17 . The non-transitory processor-executable storage medium of claim 16 , wherein the one or more convolution layers includes at least one convolutional filter that convolve a first row of the input matrix to compute a first entry in a first evaluation vector in the one or more evaluation vectors.
18 . The non-transitory processor-executable storage medium of claim 16 , wherein the operations further comprise:
applying, by a factor vector calculator connected to the one or more convolution layers, a plurality of similarity measures to the set of input values, respectively; and
computing factor vectors based on the plurality of similarity measures to form the input matrix.
19 . The non-transitory processor-executable storage medium of claim 16 , wherein the operations further comprise:
determining whether the set of input values contains the anomaly by comparing each entry in the output vector with a pre-defined threshold.
20 . The non-transitory processor-executable storage medium of claim 16 , wherein the operations further comprise:
determining one or more suggestion candidates to replace a particular input value in the set of input values based on a corresponding entry that corresponds to the particular input value in the output vector.