Adaptive anomaly detection for computer systems

1. A computer-implemented method, performed within an enterprise application operating in a virtualized environment, for collecting process data characterizing processes and connections of a virtual machine using an application-protector agent installed in said virtual machine, said computer-implemented method comprising:

collecting, by said application-protector agent, first said process data describing application processes of said virtual machine, said application-protector agent installed in said virtual machine;

extracting, by said application-protector agent, first feature vectors from the process data;

grouping, by said application-protector agent, the feature vectors to define or redefine non-anomalous clusters of feature vectors, wherein the non-anomalous clusters include respective centroids, each centroid associating with each of the plural process features a true value or a false value;

collecting, by said application-protector agent, next process data describing application processes;

extracting, by said application-protector agent, next feature vectors from the next process data;

determining, by said application-protector agent, whether or not a next feature vector is within at least one of the non-anomalous clusters; and

issuing an alert in response to a determination, by said application-protector agent, that the next feature vector is not within at least one of the non-anomalous clusters.

2. The method of claim 1 wherein:

each feature represents a respective process instance, each feature vector associating with each of plural process features a true value or a false value, each true value indicating that the respective process data indicates that the respective feature has been present in the respective process instance, each false value indicating that the respective process data indicates that the respective feature has not been present in the respective process instance.

3. The method of claim 2 further comprising subset match filtering the next feature vectors to eliminate false positives, the filtering including filtering out feature vectors for which at least one of the centroids associates true values with every process feature with which the feature vector associates a true value so that the determining applies only to feature vectors for which there is no non-anomalous cluster with a centroid that associates true values with every process feature with which that feature vector associates a true value.

4. The method of claim 1 wherein the clusters are redefined more than once to adapt to changes in the normal behavior of an application.

5. The method of claim 1 wherein the grouping re-defines existing clusters and the re-defining involves attributing greater weight to more-recent feature vectors than to less-recent feature vectors that are given a non-zero weight.

6. The method of claim 1 wherein the grouping uses an X-means algorithm to define or re-define the clusters.

7. A system including an enterprise application operating in a virtualized environment, for collecting process data characterizing processes and connections of a virtual machine using an application-protector agent installed in said virtual machine, said system further comprising non-transitory media encoded with code that, when executed by a processor, implements a method including:

collecting, by said application-protector agent, first said process data describing application processes of said virtual machine, said application-protector agent installed in said virtual machine;

extracting, by said application-protector agent, first feature vectors from the process data;

grouping, by said application-protector agent, the feature vectors to define or redefine non-anomalous clusters of feature vectors, wherein the non-anomalous clusters include respective centroids, each centroid associating with each of the plural process features a true value or a false value;

collecting, by said application-protector agent, next process data describing application processes;

extracting, by said application-protector agent, next feature vectors from the next process data;

determining, by said application-protector agent, whether or not a next feature vector is within at least one of the non-anomalous clusters; and

issuing an alert in response to a determination, by said application-protector agent, that the next feature vector is not within at least one of the non-anomalous clusters.

8. The system of claim 7 wherein:

each feature represents a respective process instance, each feature vector associating with each of plural process features a true value or a false value, each true value indicating that the respective process data indicates that the respective feature has been present in the respective process instance, each false value indicating that the respective process data indicates that the respective feature has not been present in the respective process instance.

9. The system of claim 8 wherein the method further includes subset match filtering the next feature vectors to eliminate false positives, the filtering including filtering out feature vectors for which at least one of the centroids associates true values with every process feature with which the feature vector associates a true value so that the determining applies only to feature vectors for which there is no non-anomalous cluster with a centroid that associates true values with every process feature with which that feature vector associates a true value.

10. The system of claim 8 wherein the clusters are redefined more than once to adapt to changes in the normal behavior of an application.

11. The method of claim 7 wherein the grouping re-defines existing clusters and the re-defining involves attributing greater weight to more-recent feature vectors than to less-recent feature vectors that are given a non-zero weight.

12. The method of claim 7 wherein the grouping uses an X-means algorithm to define or re-define the clusters.