Anomaly detection method using an autoencoder learning from data items collected by measuring devices
A feature value generation device includes a generator configured to generate vectors whose elements are feature values of data items collected at a plurality of timings from a target of anomaly detection, so as to normalize or standardize the vectors based on a set of predetermined vectors; a learning unit configured to learn the predetermined vectors so as to output a learning result; and a detector configured to detect, for each of the vectors normalized or standardized by the generator, an anomaly based on said each of the vectors and the learning result. The set of predetermined vectors is a set of vectors with which no anomaly is detected by the detector, and the set of vectors is updated in accordance with no anomaly being detected by the detector.
1. A feature value generation device comprising:
processing circuitry configured to
generate vectors whose elements are feature values of data items collected at each of a plurality of timings, which are each upon an elapse of a predetermined time interval that is not greater than three minutes, from a target of anomaly detection, so as to perform normalization on the vectors, after a predetermined amount of vectors are stored, based on a set of predetermined vectors, the target being a system including a plurality of measuring devices that sample the data items within the system, the feature value generation device being connected to each of the plurality of measuring devices and the processing circuitry is configured to collect the data items from the measuring devices, wherein within the predetermined time interval the feature values are collected at sub-intervals such that a plurality of data items for a particular type of data item are collected at each of the plurality of timings and the normalization of the vectors includes at each of the plurality of timings, dividing each of the collected data items for a particular type of data item by a maximum value of the particular type of data item that is collected occurs during the respective timing;
generate an autoencoder that learns the predetermined vectors so as to output a learning result by duplicating a respective vector into two, applying one of the two vectors to the input layer to the autoencoder, and applying the other of the two vectors to the output layer, to execute learning so as to output the learning result; and
for each of the plurality of timings, detect, for each of the vectors normalized, an anomaly based on said each of the vectors and the learning result,
wherein the set of predetermined vectors is a set of vectors, stored as learning data in a learning data storage, with which no anomaly is detected by the processing circuitry, and for each of the plurality of timings, the entire set of vectors stored in the learning data storage based on the most recent collected vectors when no anomaly is detected by the processing circuitry and the entire set of vectors is not stored in the learning data storage based on the most recent collected vectors when an anomaly is detected by the processing circuitry based on a most recent learning result output by the autoencoder.
2. The feature value generation device as claimed in claim 1 , wherein the system is a network that is constituted with multiple nodes being connected with each other, in which packets are transmitted and received between the nodes to provide predetermined services.
3. The feature value generation device as claimed in claim 2 , wherein the data items to be collected include MIB (Management Information Base) data, flow data by NetFlow, and CPU utilization.
4. A feature value generation method executed by a feature value generation device, the method comprising:
generating vectors whose elements are feature values of data items collected at each of a plurality of timings, which are each upon an elapse of a predetermined time interval that is not greater than three minutes, from a target of anomaly detection, so as to perform normalization on the vectors, after a predetermined amount of vectors are stored, based on a set of predetermined vectors, the target being a system including a plurality of measuring devices that sample the data items within the system, the feature value generation device being connected to each of the plurality of measuring devices and method includes collecting the data items from the measuring devices, wherein within the predetermined time interval the feature values are collected at sub-intervals such that a plurality of data items for a particular type of data item are collected at each of the plurality of timings and the normalization of the vectors includes at each of the plurality of timings, dividing each of the collected data items for a particular type of data item by a maximum value of the particular type of data item that is collected occurs during the respective timing;
generating an autoencoder that learns the predetermined vectors so as to output a learning result by duplicating a respective vector into two, applying one of the two vectors to the input layer to the autoencoder, and applying the other of the two vectors to the output layer, to execute learning so as to output the learning result; and
for each of the plurality of timings, detecting, for each of the vectors normalized, an anomaly based on said each of the vectors and the learning result,
wherein the set of predetermined vectors is a set of vectors, stored as learning data in a learning data storage, with which no anomaly is detected, and for each of the plurality of timings, the entire set of vectors stored in the learning data storage based on the most recent collected vectors when no anomaly is detected and the entire set of vectors is not stored in the learning data storage based on the most recent collected vectors when an anomaly is detected based on a most recent learning result output by the autoencoder.
5. A non-transitory computer-readable recording medium having a program stored thereon for causing a computer to execute the feature value generation method as claimed in claim 4 .