Systems and methods for detecting anomalous system or network behavior
A system and associated methods for the detection of anomalous behavior in a system. In some embodiments, time-series data that is obtained from the system (such as log data) may be used as an input to a process that converts the data into greyscale values. The greyscale values are used to construct an “image” of the system operation that is used as an input to a convolutional neural network (CNN). The image is used to train the neural network so that the neural network is able to recognize when other input “images” constructed from time-series data are anomalous or otherwise indicative of a difference between the prior (and presumed normal or acceptable) and the current operation of the system.
1. A method for detecting anomalous behavior in a system, comprising:
acquiring a set of time-series data generated by or characterizing past behavior of the system;
converting the set of time-series data into a set of grey-scale values of the past behavior of the system;
forming a plurality of matrices from the set of grey-scale values of the past behavior of the system, wherein each matrix represents the grey-scale values at a specific time or during a specific time interval as an image;
training a neural network using the plurality of matrices of the past behavior of the system to determine a predicted state vector of the system at a future time, the future time determined by system entropy or causal correlation between events that occur in the system;
acquiring time-series data representing a current operational state of the system;
converting the set of time-series data representing the current operational state into grey-scale values of the current operational state;
forming a matrix from the grey-scale values of the current operational state;
inputting the formed matrix of the current operational state to the trained neural network, the neural network trained to:
determine a predicted state vector of the system based on the formed matrix of the current operational state;
compare the predicted state vector of the system at a predicted time to the actual state vector of the system at the predicted time;
identify a difference between the predicted state vector and the actual state vector of the system at the predicted time;
generate an output from the trained neural network that includes an indication that the predicted state vector and the actual state vector are different at the predicted time; and
determining that the output indicates the presence of an anomaly in the system based on the indication that the predicted state vector and the actual state vector are different at the predicted time; and
generating an alert in response to the indication of the presence of the anomaly.
2. The method of claim 1 , wherein the time-series data of the past behavior of the system represents a value of a parameter or a characteristic of the system at a specific time or during a specific time interval in the past.
3. The method of claim 1 , wherein the neural network is a convolutional neural network (CNN).
4. The method of claim 1 , wherein the set of time-series data of the past behavior of the system is one of data representing TCP/IP traffic, GPU or CPU utilization, hard disk read/write actions, or memory utilization.
5. The method of claim 1 , wherein the neural network is further trained to determine a probability that the system is operating in a normal state, a probability that a system anomaly has occurred, or an image describing a future state of the system.
6. The method of claim 5 , wherein the output represents the probability that the system is operating in a normal state, and determining if the output indicates the presence of an anomaly further comprises determining if the probability is less than a threshold value.
7. The method of claim 1 , wherein the system is a network of nodes with a connection between the nodes.
8. An apparatus for detecting anomalous behavior in a system, comprising:
an electronic processor programmed with a set of executable instructions;
an electronic data storage in which are stored the set of executable instructions, wherein the set of instructions includes instructions, which when executed, cause the apparatus to implement one or more processes to
acquire a set of time-series data generated by or characterizing past behavior of the system;
convert the set of time-series data into a set of grey-scale values of the past behavior of the system;
form a plurality of matrices from the set of grey-scale values of the past behavior of the system, wherein each matrix represents the grey-scale values at a specific time or during a specific time interval as an image;
train a neural network using the plurality of matrices of the past behavior of the system to determine a predicted state vector of the system at a future time, the future time determined by system entropy or causal correlation between events that occur in the system;
acquire time-series data representing a current operational state of the system;
convert the set of time-series data representing the current operational state into grey-scale values of the current operational state;
form a matrix from the grey-scale values of the current operational state;
input the formed matrix of the current operational state to the trained neural network, the neural network trained to:
determine a predicted state vector of the system based on the formed matrix of the current operational state;
compare the predicted state vector of the system at a predicted time to the actual state vector of the system at the predicted time;
identify a difference between the predicted state vector and the actual state vector of the system at the predicted time;
generate an output from the trained neural network that includes an indication that the predicted state vector and the actual state vector are different at the predicted time;
determine that the output indicates the presence of an anomaly in the system based on the indication that the predicted state vector and the actual state vector are different at the predicted time; and
generate an alert in response to the indication of the presence of the anomaly.
9. The apparatus of claim 8 , wherein the time series data of the past behavior of the system represents a value of a parameter or a characteristic of the system at a specific time or during a specific time interval in the past.
10. The apparatus of claim 8 , wherein the neural network is a convolutional neural network (CNN).
11. The apparatus of claim 8 , wherein the set of time-series data of the past behavior of the system is one of data representing TCP/IP traffic, GPU or CPU utilization, hard disk read/write actions, or memory utilization.
12. The apparatus of claim 8 , wherein the neural network is further trained to determine a probability that the system is operating in a normal state, a probability that a system anomaly has occurred, or an image describing a future state of the system.
13. The apparatus of claim 12 , wherein the output represents the probability that the system is operating in a normal state, and determining if the output indicates the presence of an anomaly further comprises determining if the probability is less than a threshold value.
14. The apparatus of claim 8 , wherein the system is a network of nodes with a connection between the nodes.