Multi-baseline unsupervised security-incident and network behavioral anomaly detection in cloud-based compute environments

1. A method for detecting anomalous network activity in a cloud-based compute environment, comprising:

receiving configuration data and network activity observations for a set of virtual entities hosted in the cloud-based compute environment;

creating a profile for each virtual entity in the set of virtual entities based on the configuration data, when the virtual entity does not already have an associated profile;

creating a virtual entity group profile for a group of the virtual entities, wherein the group of virtual entities (a) implement a same application or a same functionality and (b) have a same expected network behavior;

dynamically updating the profile of each virtual entity and the virtual entity group profile with the respective network activity observations of the virtual entity; and

determining whether anomalies have been detected in the set of virtual entities based on the profiles of the virtual entities and the virtual entity group profile; and

wherein security system uses probabilistic unsupervised learning methods to identify expected network behavior and deviations from expected profiles, and updated profiles are checked to see if they include significant deviations from previous profiles, such that values or changes in values exceed thresholds and the significance of the deviations take into account both the differences between expected and actual numeric values, and uncertainty in expected values and measurement of actual observations.

2. The method of claim 1 , wherein

the set of virtual entities includes one or more virtual machines or containers hosted on one or more physical machines.

3. The method of claim 1 , wherein creating the profile further comprises:

creating a connections group when a virtual entity in the set of virtual entities is identified as having similar network behavior with the connections group.

4. The method of claim 3 , wherein creating the profile further comprises:

including a set of probabilistic distributions over values of a large set of factors, wherein the factors represent an aspect of the behavior of the virtual entity.

5. The method of claim 4 , wherein

the factors of the large set of factors include observable and unobservable factors, and the factors may be learned from the probabilistic dependencies on observable factors.

6. The method of claim 5 , wherein

the aggregated learned distribution of values of all the factors represents a modeled baseline of the virtual entity's observed behavior or internal state.

7. The method of claim 6 , wherein determining whether anomalies have been detected further comprises:

checking the updated profiles to determine if significant deviations in values exceed a threshold of normal virtual entity behavior.

8. The method of claim 7 , wherein

checking whether the significance of the deviations takes into account both the difference between the expected and actual numeric values of a factor, and the uncertainty in the expected values and the uncertainty in the measurement of the actual observation.

9. The method of claim 1 , further comprising:

reporting the anomaly, when an anomaly has been detected; and

taking a mitigating activity, when an anomaly has been detected.

10. A system for detecting anomalous network activity in a based compute environment, comprising:

a processing circuitry; and

a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to:

receive configuration data and network activity observations for a set of virtual entities hosted in the cloud-based compute environment;

create a profile for each virtual entity in the set of virtual entities based on the configuration data, when the virtual entity does not already have an associated profile;

create a virtual entity group profile for a group of the virtual entities,

wherein

the group of virtual entities (a) implement a same application or a same functionality and (b) have a same expected network behavior;

dynamically update the profile of each virtual entity and the virtual entity group profile with the respective network activity observations of the virtual entity; and

determine whether anomalies have been detected in the set of virtual entities based on the profiles of the virtual entities and the virtual entity group profile; and

wherein security system uses probabilistic unsupervised learning methods to identify expected network behavior and deviations from expected profiles, and updated profiles are checked to see if they include significant deviations from previous profiles, such that values or changes in values exceed thresholds and the significance of the deviations take into account both the differences between expected and actual numeric values, and uncertainty in expected values and measurement of actual observations.

11. The system of claim 10 , wherein

the set of virtual entities includes one or more virtual machines or containers hosted on one or more physical machines.

12. The system of claim 10 , wherein the system is further configured to:

create a connections group when a virtual entity in the set of virtual entities is identified as having similar network behavior with the connections group.

13. The system of claim 12 , wherein the system is further configured to:

comprise a set of probabilistic distributions over values of a large set of factors, wherein the factors represent an aspect of the behavior of the virtual entity.

14. The system of claim 13 , wherein

the factors of the large set of factors include observable and unobservable factors, and the factors may be learned from the probabilistic dependencies on observable factors.

15. The system of claim 14 , wherein

the aggregated learned distribution of values of all the factors represents a modeled baseline of the virtual entity's observed behavior or internal state.

16. The system of claim 15 , wherein the system is further configured to:

check the updated profiles to see if significant deviations in values exceed a threshold of normal virtual entity behavior.

17. The system of claim 16 , wherein

checking whether the significance of the deviations takes into account both the difference between the expected and actual numeric values of a factor, and the uncertainty in the expected values and the uncertainty in the measurement of the actual observation.

18. The system of claim 10 , wherein the system is further configured to:

report the anomaly, when an anomaly has been detected; and

take a mitigating activity, when an anomaly has been detected.

19. The system of claim 18 , wherein the system is further configured to:

block a virtual entity associated with the anomaly.

20. A non-transitory computer readable medium having stored thereon instructions for causing processing circuitry to perform a process for detecting anomalous network activity in a cloud-based compute environment, the process comprising:

receiving configuration data and network activity observations for a set of virtual entities hosted in the cloud-based compute environment;

creating a profile for each virtual entity in the set of virtual entities based on the configuration data, when the virtual entity does not already have an associated profile;

creating a virtual entity group profile for a group of the virtual entities, wherein the group of virtual entities (a) implement a same application or a same functionality and (b) have a same expected network behavior;

dynamically updating the profile of each virtual entity and the virtual entity group profile with the respective network activity observations of the virtual entity; and

determining whether anomalies have been detected in the set of virtual entities based on the profiles of the virtual entities and the virtual entity group profile; and

wherein the security system uses probabilistic unsupervised learning methods to identify expected network behavior and deviations from expected profiles, and updated profiles are checked to see if they include significant deviations from previous profiles, such that values or changes in values exceed thresholds and the significance of the deviations take into account both the differences between expected and actual numeric values, and uncertainty in expected values and measurement of actual observations.