Detecting anomalies in distributed applications based on process data
Techniques for detecting anomalies in a distributed application based on process data are provided. This process data can include, e.g., the hierarchy (i.e., tree) of processes created and run by the application, the file system operations performed by each process, the network access operations performed by each process.
1. A method comprising:
receiving, by a computer system, a first plurality of process data records from a computing cluster running a distributed application, the distributed application being composed of a workload comprising one or more pods, the first plurality of process data records including first information regarding software processes running within the one or more pods over a first time period;
building, by the computer system, a reference model based on the first plurality of process data records, the reference model capturing normal process behavior of the workloads and the one or more pods;
receiving, by the computer system, a second plurality of process data records from the computing cluster, the second plurality of process data records including second information regarding software processes running within the one or more pods over a second time period;
comparing, by the computer system, the second plurality of process data records to the reference model;
upon detecting a deviation between the second plurality of process data records and the reference model, generating, by the computer system, a record indicating an anomaly in the workload or the one or more pods during the second time period; and
adjusting the resource allocation or restarting the one or more pods exhibiting the anomaly to restore normal operation of the distributed application.
2. The method of claim 1 wherein the first and second pluralities of process data records include one or more process tree records, each process tree record including an identifier of a software process running within a pod and an identifier of a parent process of the software process, if the software process was spawned or forked from the parent process.
3. The method of claim 1 wherein the first and second pluralities of process data records include one or more file system operation records, each file system operation record identifying a file system operation performed by a software process running within a pod and including metadata regarding the file system operation.
4. The method of claim 1 wherein the first and second pluralities of process data records include one or more network operation records, each network operation record identifying a network operation performed by a software process running within a pod and including metadata regarding the network operation.
5. The method of claim 1 wherein the building comprises:
transforming the first plurality of process data records into a format for building the reference model; and
building the reference model using the transformed first plurality of process data records.
6. The method of claim 1 wherein the comparing comprises:
transforming the second plurality of process data records into a format for comparison to the reference model; and
comparing the transformed second plurality of process data records to the reference model.
7. The method of claim 1 further comprising, for each pod of the distributed application that is part of a replica set:
comparing a first subset of the second plurality of process data records corresponding to the pod to second subsets of the second plurality of process data records corresponding to other pods in the replica set; and
upon detecting a deviation between the first subset and the second subsets, generating another record indicating an anomaly in the pod during the second time period.
8. A non-transitory computer readable storage medium having stored thereon program code executable by a computer system, the method comprising:
receiving a first plurality of process data records from a computing cluster running a distributed application, the distributed application being composed of a workload comprising one or more pods, the first plurality of process data records including first information regarding software processes running within the one or more pods over a first time period;
building a reference model based on the first plurality of process data records, the reference model capturing normal process behavior of the workloads and the one or more pods;
receiving a second plurality of process data records from the computing cluster, the second plurality of process data records including second information regarding software processes running within the one or more pods over a second time period;
comparing the second plurality of process data records to the reference model;
upon detecting a deviation between the second plurality of process data records and the reference model, generating a record indicating an anomaly in the workload or the one or more pods during the second time period; and
adjusting the resource allocation or restarting the one or more pods exhibiting the anomaly to restore normal operation of the distributed application.
9. The non-transitory computer readable storage medium of claim 8 wherein the first and second pluralities of process data records include one or more process tree records, each process tree record including an identifier of a software process running within a pod and an identifier of a parent process of the software process, if the software process was spawned or forked from the parent process.
10. The non-transitory computer readable storage medium of claim 8 wherein the first and second pluralities of process data records include one or more file system operation records, each file system operation record identifying a file system operation performed by a software process running within a pod and including metadata regarding the file system operation.
11. The non-transitory computer readable storage medium of claim 8 wherein the first and second pluralities of process data records include one or more network operation records, each network operation record identifying a network operation performed by a software process running within a pod and including metadata regarding the network operation.
12. The non-transitory computer readable storage medium of claim 8 wherein the building comprises:
transforming the first plurality of process data records into a format suitable for building the reference model; and
building the reference model using the transformed first plurality of process data records.
13. The non-transitory computer readable storage medium of claim 8 wherein the comparing comprises:
transforming the second plurality of process data records into a format for comparison to the reference model; and
comparing the transformed second plurality of process data records to the reference model.
14. The non-transitory computer readable storage medium of claim 8 wherein the method further comprises, for each pod of the distributed application that is part of a replica set:
comparing a first subset of the second plurality of process data records corresponding to the pod to second subsets of the second plurality of process data records corresponding to other pods in the replica set; and
upon detecting a deviation between the first subset and the second subsets, generating another record indicating an anomaly in the pod during the second time period.
15. A computer system comprising:
a processor; and
a non-transitory computer readable medium having stored thereon program code that, when executed, causes the processor to:
receive a first plurality of process data records from a computing cluster running a distributed application, the distributed application being composed of a workload comprising one or more pods, the first plurality of process data records including first information regarding software processes running within the one or more pods over a first time period;
build a reference model based on the first plurality of process data records, the reference model capturing normal process behavior of the workloads and the one or more pods;
receive a second plurality of process data records from the computing cluster, the second plurality of process data records including second information regarding software processes running within the one or more pods over a second time period;
compare the second plurality of process data records to the reference model;
upon detecting a deviation between the second plurality of process data records and the reference model, generate a record indicating an anomaly in the workload or the one or more pods during the second time period; and
adjust the resource allocation or restarting the one or more pods exhibiting the anomaly to restore normal operation of the distributed application.
16. The computer system of claim 15 wherein the first and second pluralities of process data records include one or more process tree records, each process tree record including an identifier of a software process running within a pod and an identifier of a parent process of the software process, if the software process was spawned or forked from the parent process.
17. The computer system of claim 15 wherein the first and second pluralities of process data records include one or more file system operation records, each file system operation record identifying a file system operation performed by a software process running within a pod and including metadata regarding the file system operation.
18. The computer system of claim 15 wherein the first and second pluralities of process data records include one or more network operation records, each network operation record identifying a network operation performed by a software process running within a pod and including metadata regarding the network operation.
19. The computer system of claim 15 wherein the building comprises:
transforming the first plurality of process data records into a format for building the reference model; and
building the reference model using the transformed first plurality of process data records.
20. The computer system of claim 15 wherein the comparing comprises:
transforming the second plurality of process data records into a format for comparison to the reference model; and
comparing the transformed second plurality of process data records to the reference model.
21. The computer system of claim 15 wherein the program code further causes the processor to, for each pod of the distributed application that is part of a replica set:
compare a first subset of the second plurality of process data records corresponding to the pod to second subsets of the second plurality of process data records corresponding to other pods in the replica set; and
upon detecting a deviation between the first subset and the second subsets, generate another record indicating an anomaly in the pod during the second time period.