HIERARCHICAL NOVELTY DETECTION USING INTENDED STATES FOR NETWORK SECURITY
The disclosure provides an approach for detecting and preventing attacks in a network. Embodiments include determining a plurality of network behaviors of a process by monitoring the process. Embodiments include generating a plurality of intended states for the process based on subsets of the plurality of network behaviors. Embodiments include determining a plurality of intended state clusters by applying a clustering technique to the plurality of intended states. Embodiments include determining a state of the process. Embodiments include identifying a given cluster of the plurality of intended state clusters that corresponds to the state of the process. Embodiments include selecting a novelty detection technique based on a size of the given cluster. Embodiments include using the novelty detection technique to determine, based on the given cluster and the state of the process, whether to generate a security alert for the process.
1 . A method of detecting and preventing attacks in a network, comprising:
determining a plurality of network behaviors of a process;
generating a plurality of intended states for the process based on subsets of the plurality of network behaviors;
determining a plurality of intended state clusters by applying a clustering technique to the plurality of intended states;
determining a state of the process;
identifying a given cluster of the plurality of intended state clusters that corresponds to the state of the process;
selecting a novelty detection technique based on a size of the given cluster; and
using the novelty detection technique to determine, based on the given cluster and the state of the process, whether to perform one or more security threat prevention actions.
2 . The method of claim 1 , wherein generating the plurality of intended states of the process comprises extracting features from the subsets of the plurality of network behaviors to produce feature vectors.
3 . The method of claim 2 , wherein applying the clustering technique to the plurality of intended states comprises applying k-modes or k-means clustering to the feature vectors.
4 . The method of claim 1 , wherein identifying the given cluster of the plurality of intended state clusters that corresponds to the state of the process comprises comparing the state of the process to a reference point of the given cluster.
5 . The method of claim 1 , wherein the novelty detection technique is selected from:
a tree-based model;
weighted hamming distances; or
review by a user.
6 . The method of claim 1 , wherein a security alert is generated if the novelty detection technique indicates that the state of the process is an anomaly.
7 . The method of claim 1 , wherein the state of the process comprises a feature vector indicating one or more of:
the process did or did not make outbound public address access;
the process did or did not make outbound private address access;
the process did or did not make an outbound connection on an ephemeral port;
the process did or did not make an outbound connection on an a well-known port;
the process did or did not receive an inbound connection on an ephemeral port;
the process did or did not receive an inbound connection on an a well-known port;
the process did or did not make an outbound connection on an a specific port; or
the process did or did not receive an inbound connection on a particular port.
8 . A system, comprising: one or more processors; and a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the system to perform a method of detecting and preventing attacks in a network, the method comprising:
determining a plurality of network behaviors of a process;
generating a plurality of intended states for the process based on subsets of the plurality of network behaviors;
determining a plurality of intended state clusters by applying a clustering technique to the plurality of intended states;
determining a state of the process;
identifying a given cluster of the plurality of intended state clusters that corresponds to the state of the process;
selecting a novelty detection technique based on a size of the given cluster; and
using the novelty detection technique to determine, based on the given cluster and the state of the process, whether to perform one or more security threat prevention actions.
9 . The system of claim 8 , wherein generating the plurality of intended states of the process comprises extracting features from the subsets of the plurality of network behaviors to produce feature vectors.
10 . The system of claim 9 , wherein applying the clustering technique to the plurality of intended states comprises applying k-modes or k-means clustering to the feature vectors.
11 . The system of claim 8 , wherein identifying the given cluster of the plurality of intended state clusters that corresponds to the state of the process comprises comparing the state of the process to a reference point of the given cluster.
12 . The system of claim 8 , wherein the novelty detection technique is selected from:
a tree-based model;
weighted hamming distances; or
review by a user.
13 . The system of claim 8 , wherein a security alert is generated if the novelty detection technique indicates that the state of the process is an anomaly.
14 . The system of claim 8 , wherein the state of the process comprises a feature vector indicating one or more of:
the process did or did not make outbound public address access;
the process did or did not make outbound private address access;
the process did or did not make an outbound connection on an ephemeral port;
the process did or did not make an outbound connection on an a well-known port;
the process did or did not receive an inbound connection on an ephemeral port;
the process did or did not receive an inbound connection on an a well-known port;
the process did or did not make an outbound connection on an a specific port; or
the process did or did not receive an inbound connection on a particular port.
15 . A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a computing system, cause the computing system to perform a method of detecting and preventing attacks in a network, the method comprising:
determining a plurality of network behaviors of a process;
generating a plurality of intended states for the process based on subsets of the plurality of network behaviors;
determining a plurality of intended state clusters by applying a clustering technique to the plurality of intended states;
determining a state of the process;
identifying a given cluster of the plurality of intended state clusters that corresponds to the state of the process;
selecting a novelty detection technique based on a size of the given cluster; and
using the novelty detection technique to determine, based on the given cluster and the state of the process, whether to perform one or more security threat prevention actions.
16 . The non-transitory computer-readable medium of claim 15 , wherein generating the plurality of intended states of the process comprises extracting features from the subsets of the plurality of network behaviors to produce feature vectors.
17 . The non-transitory computer-readable medium of claim 16 , wherein applying the clustering technique to the plurality of intended states comprises applying k-modes or k-means clustering to the feature vectors.
18 . The non-transitory computer-readable medium of claim 15 , wherein identifying the given cluster of the plurality of intended state clusters that corresponds to the state of the process comprises comparing the state of the process to a reference point of the given cluster.
19 . The non-transitory computer-readable medium of claim 15 , wherein the novelty detection technique is selected from:
a tree-based model;
weighted hamming distances; or
review by a user.
20 . The non-transitory computer-readable medium of claim 15 , wherein a security alert is generated if the novelty detection technique indicates that the state of the process is an anomaly.