Metric space modeling of network communication
Devices and techniques are generally described for detection of network anomalies. In various examples, first data describing network communication between a plurality of source entities and a plurality of destination entities may be received. In some examples, respective feature data representing network communication between a respective source entity and one or more of the plurality of destination entities may be generated. In some examples, an unsupervised machine learning model may be used to determine a first number of clusters of the feature data. In various cases, a first source entity that is an outlier with respect to the first number of clusters may be determined based at least in part on the first number of clusters. The first source entity may be classified as an anomalous entity.
1. A computer-implemented method of metric space modeling of network communication, the method comprising:
receiving input data representing network interactions between a plurality of source entities and a plurality of destination entities over a first time period;
determining, for a first source entity of the plurality of source entities, a first feature vector, the first feature vector comprising a plurality of elements, wherein each element of the plurality of elements is associated with a respective destination entity of the plurality of destination entities, and wherein a value of each of the elements represents respective metric values describing network communication between the first source entity and the respective destination entity;
inputting a plurality of feature vectors into a clustering algorithm, the plurality of feature vectors including the first feature vector, wherein each feature vector of the plurality of feature vectors represents network interactions of one of the plurality of source entities;
generating, using the clustering algorithm, a first number of clusters of the plurality of feature vectors;
determining that the first feature vector is not included in any clusters of the first number of clusters; and
classifying the first source entity as an anomalous network entity based at least in part on the first feature vector being an outlier with respect to the first number of clusters.
2. The computer-implemented method of claim 1 , wherein each of the respective metric values represents at least one of: a number of packets sent between the first source entity and the respective destination entity; a duration of time during which the first source entity communicated with the respective destination entity; an amount of data exchanged between the first source entity and the respective destination entity; and a number of connections established to a port of the respective destination entity.
3. The computer-implemented method of claim 1 , further comprising:
identifying a first element of the plurality of feature vectors that corresponds to a first internet protocol (IP) address of a first destination entity;
identifying a second element of the plurality of feature vectors that corresponds to a second IP address of a second destination entity;
determining that the first IP address and the second IP address resolve to the same domain; and
representing metric values for the first IP address and the second IP address in a single element of the plurality of feature vectors.
4. A method comprising:
receiving first data describing network communication between a plurality of source entities and a plurality of destination entities;
generating, for each source entity of the plurality of source entities, respective feature data representing network communication between the respective source entity and one or more of the plurality of destination entities, the respective feature data representing network communication for the respective source entity over a first time period;
determining, using an unsupervised machine learning model, a first number of clusters of the feature data;
determining, based at least in part on the first number of clusters, a first source entity associated with feature data that is not included in a cluster of the first number of clusters; and
classifying the first source entity as an anomalous entity.
5. The method of claim 4 , further comprising:
generating for a first source entity of the plurality of source entities, first feature data comprising a first feature vector, wherein the first feature vector includes a respective element for each destination entity of the plurality of destination entities, wherein the first feature vector further includes a value for each respective element, the value being associated with network communication between the first source entity and the respective destination entity associated with the respective element.
6. The method of claim 5 , further comprising reducing a dimensionality of the first feature vector to generate a second feature vector, wherein the dimensionality is reduced based at least in part on a similarity between two or more destination entities of the plurality of destination entities.
7. The method of claim 4 , further comprising:
determining first feature data representing a first source entity of the plurality of source entities;
determining second feature data representing a second source entity of the plurality of source entities; and
determining, using a first distance function, a distance between the first feature data and the second feature data in a feature space of the first feature data and the second feature data, wherein the first feature data and the second feature data are included in a cluster of the first number of clusters based at least in part on the distance.
8. The method of claim 4 , further comprising:
determining first feature data representing a first source entity of the plurality of source entities;
determining second feature data representing a second source entity of the plurality of source entities;
generating, using a hash function, a first hash value representing the first feature data; and
generating, using the hash function, a second hash value representing the second feature data, wherein the determining the first number of clusters is based at least in part on grouping the first hash value and the second hash value together in a cluster of the first number of clusters.
9. The method of claim 4 , wherein:
the respective feature data comprises a vector representation of the network communication for each source entity of the plurality of source entities; and
each element of the vector representation represents one or more of a number of packets sent, an amount of data sent, a number of times a communication channel was established, and a duration of time during which a communication channel was established with a respective destination entity of the plurality of destination entities.
10. The method of claim 4 , further comprising:
determining, based at least in part on the first number of clusters, a second source entity that is not included in any cluster of the first number of clusters;
determining a similarity between feature data representing the second source entity and feature data representing a first cluster, wherein the network communication associated with the first cluster has been classified as valid network communication; and
classifying network communication associated with the second source entity as valid based at least in part on the similarity.
11. The method of claim 4 , further comprising:
determining a first element of the feature data associated with a first internet protocol (IP) address;
determining a second element of the feature data associated with a second IP address different from the first IP address;
determining that the first IP address and the second IP address are associated with the same domain; and
reducing a dimensionality of the feature data based at least in part on the first IP address and the second IP address being associated with the same domain.
12. The method of claim 4 , further comprising receiving, by the unsupervised machine learning model, a configuration file comprising a threshold number of source entities for a given cluster, a definition of a distance function, and a distance threshold used for outlier detection.
13. A system comprising:
at least one processor; and
non-transitory computer-readable memory storing instructions that, when executed by the at least one processor, are effective to:
receive first data describing network communication between a plurality of source entities and a plurality of destination entities;
generate, for each source entity of the plurality of source entities, respective feature data representing network communication between the respective source entity and one or more of the plurality of destination entities, the respective feature data representing network communication for the respective source entity over a first time period;
determine, using an unsupervised machine learning model, a first number of clusters of the feature data;
determine based at least in part on the first number of clusters, a first source entity associated with feature data that is not included in a cluster of the first number of clusters; and
classify the first source entity as an anomalous entity.
14. The system of claim 13 , the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:
generate for a first source entity of the plurality of source entities, first feature data comprising a first feature vector, wherein the first feature vector includes a respective element for each destination entity of the plurality of destination entities, wherein the first feature vector further includes a value for each respective element, the value being associated with network communication between the first source entity and the respective destination entity associated with the respective element.
15. The system of claim 14 , the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:
reduce a dimensionality of the first feature vector to generate a second feature vector, wherein the dimensionality is reduced based at least in part on a similarity between two or more destination entities of the plurality of destination entities.
16. The system of claim 13 , the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:
determine first feature data representing a first source entity of the plurality of source entities;
determine second feature data representing a second source entity of the plurality of source entities; and
determine, using a first distance function, a distance between the first feature data and the second feature data in a feature space of the first feature data and the second feature data, wherein the first feature data and the second feature data are included in a cluster of the first number of clusters based at least in part on the distance.
17. The system of claim 13 , the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:
determine first feature data representing a first source entity of the plurality of source entities;
determine second feature data representing a second source entity of the plurality of source entities;
generate, using a hash function, a first hash value representing the first feature data; and
generate, using the hash function, a second hash value representing the second feature data, wherein the determining the first number of clusters is based at least in part on grouping the first hash value and the second hash value together in a cluster of the first number of clusters.
18. The system of claim 13 , wherein:
the respective feature data comprises a vector representation of the network communication for each source entity of the plurality of source entities; and
each element of the vector representation represents one or more of a number of packets sent, an amount of data sent, a number of times a communication channel was established, and a duration of time during which a communication channel was established with a respective destination entity of the plurality of destination entities.
19. The system of claim 13 , the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:
determine, based at least in part on the first number of clusters, a second source entity that is not included in any cluster of the first number of clusters;
determine a similarity between feature data representing the second source entity and feature data representing a first cluster, wherein the network communication associated with the first cluster has been classified as valid network communication; and
classify network communication associated with the second source entity as valid based at least in part on the similarity.
20. The system of claim 13 , the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:
determine a first element of the feature data associated with a first internet protocol (IP) address;
determine a second element of the feature data associated with a second IP address different from the first IP address;
determine that the first IP address and the second IP address are associated with the same domain; and
reduce a dimensionality of the feature data based at least in part on the first IP address and the second IP address being associated with the same domain.