Detecting anomalies in a distributed application
Anomalies are detected in a distributed application that runs on a plurality of nodes to execute at least first and second workloads. The method of detecting anomalies includes collecting first network traffic data of the first workload and second network traffic data of the second workload during a first period of execution of the first and second workloads, collecting third network traffic data of the first workload and fourth network traffic data of the second workload during a second period of execution of the first and second workloads, and detecting an anomaly in the distributed application based on a comparison of the third network traffic data against the first network traffic data or a comparison of the fourth network traffic data against the second network traffic data. Anomalies may also be detected by comparing network traffic data of two groups of containers executing the same workload.
1. A method of detecting anomalies in a distributed application that runs on a plurality of nodes to execute a plurality of workloads, comprising:
collecting first network traffic data of the workloads at a number of different times during a first period of execution of the workloads, the first network traffic data including egress traffic data, ingress traffic data, and inter-workload traffic data;
collecting second network traffic data of the workloads at a number of different times during a second period of execution of the workloads after the first period of execution, the second network traffic data including egress traffic data, ingress traffic data, and inter-workload traffic data; and
detecting an anomaly in the distributed application based on a comparison of a frequency of private IP addresses specified in the egress traffic data in the collected first network traffic data and a frequency of private IP addresses specified in the egress traffic data in the collected second network traffic data.
2. The method of claim 1 , further comprising:
detecting an anomaly in the distributed application based on a comparison of a port, protocol, or domain specified in the egress traffic data in the collected first network traffic data and a port, protocol, or domain specified in the egress traffic data in the collected second network traffic data.
3. The method of claim 1 , further comprising:
determining a first data rate to all workload IDs specified in the egress traffic data in the collected first network traffic data and a second data rate to all workload IDs specified in the egress traffic data in the collected second network traffic data; and
detecting an anomaly in the distributed application based on a comparison of the first data rate and the second data rate.
4. The method of claim 1 , further comprising:
detecting an anomaly in the distributed application based on the ingress traffic data in the collected first network traffic data and the ingress traffic data in the collected second network traffic data.
5. The method of claim 4 , wherein:
the anomaly is detected in the distributed application based on a comparison of a frequency of private IP addresses specified in the ingress traffic data in the collected first network traffic data and a frequency of private IP addresses specified in the ingress traffic data in the collected second network traffic data.
6. The method of claim 4 , further comprising:
determining a first data rate to all workload IDs specified in the ingress traffic data in the collected first network traffic data and a second data rate to all workload IDs specified in the ingress traffic data in the collected second network traffic data, wherein
the anomaly is detected in the distributed application based on a comparison of the first data rate and the second data rate.
7. The method of claim 1 , further comprising:
detecting an anomaly in the distributed application based on the inter-workload traffic data in the collected first network traffic data and the inter-workload traffic data in the collected second network traffic data.
8. The method of claim 7 , further comprising:
determining a first data rate to a particular workload ID specified in the inter-workload traffic data in the collected first network traffic data, and a second data rate to the particular workload ID specified in the inter-workload traffic data in the collected second network traffic data, wherein
the anomaly is detected in the distributed application based on a comparison of the first data rate and the second data rate.
9. The method of claim 7 , further comprising:
determining a first data rate to all workload IDs specified in the inter-workload traffic data in the collected first network traffic data and a second data rate to all workload IDs specified in the inter-workload traffic data in the collected second network traffic data, wherein
the anomaly is detected in the distributed application based on a comparison of the first data rate and the second data rate.
10. A computing system for detecting anomalies in a distributed application that runs on a plurality of nodes to execute a plurality of workloads, said computing system comprising:
a storage device in which first network traffic data of the workloads at a number of different times during a first period of execution of the workloads and second network traffic data of the workloads at a number of different times during a second period of execution of the workloads after the first period of execution, are collected, wherein the first network traffic data includes egress traffic data, ingress traffic data, and inter-workload traffic data and the second network traffic data includes egress traffic data, ingress traffic data, and inter-workload traffic data; and
an anomaly detection server configured to detect an anomaly in the distributed application based on a comparison of a frequency of private IP addresses specified in the ingress traffic data in the collected first network traffic data and a frequency of private IP addresses specified in the ingress traffic data in the collected second network traffic data.
11. The computing system of claim 10 , wherein the anomaly detection server is further configured to:
detect an anomaly in the distributed application based on the egress traffic data in the collected first network traffic data and the egress traffic data in the collected second network traffic data.
12. The computing system of claim 11 , wherein
the anomaly is detected in the distributed application based on a comparison of a port, protocol, or domain specified in the egress traffic data in the collected first network traffic data and a port, protocol, or domain specified in the egress traffic data in the collected second network traffic data.
13. The computing system of claim 11 , wherein
the anomaly is detected in the distributed application based on a comparison of a frequency of private IP addresses specified in the egress traffic data in the collected first network traffic data and a frequency of private IP addresses specified in the egress traffic data in the collected second network traffic data.
14. The computing system of claim 11 , wherein the anomaly detection server is further configured to:
determine a first data rate to all workload IDs specified in the egress traffic data in the collected first network traffic data and a second data rate to all workload IDs specified in the egress traffic data in the collected second network traffic data, wherein
the anomaly is detected in the distributed application based on a comparison of the first data rate and the second data rate.
15. The computing system of claim 10 , wherein the anomaly detection server is further configured to:
determine a first data rate to all workload IDs specified in the ingress traffic data in the collected first network traffic data and a second data rate to all workload IDs specified in the ingress traffic data in the collected second network traffic data; and
detect an anomaly in the distributed application based on a comparison of the first data rate and the second data rate.
16. The computing system of claim 10 , wherein the anomaly detection server is further configured to:
detect an anomaly in the distributed application based on the inter-workload traffic data in the collected first network traffic data and the inter-workload traffic data in the collected second network traffic data.
17. The computing system of claim 16 , wherein the anomaly detection server is further configured to:
determine a first data rate to a particular workload ID specified in the inter-workload traffic data in the collected first network traffic data, and a second data rate to the particular workload ID specified in the inter-workload traffic data in the collected second network traffic data, wherein
the anomaly is detected in the distributed application based on a comparison of the first data rate and the second data rate.
18. The computing system of claim 16 , wherein the anomaly detection server is further configured to:
determine a first data rate to all workload IDs specified in the inter-workload traffic data in the collected first network traffic data and a second data rate to all workload IDs specified in the inter-workload traffic data in the collected second network traffic data, wherein
the anomaly is detected in the distributed application based on a comparison of the first data rate and the second data rate.