System and method for monitoring application security in a network environment

1. A method executing in an application aware security module, comprising:

detecting an application role in a plurality of application roles of a distributed application executing in a plurality of virtual machines in a network environment;

generating a role profile for the application role from an interaction pattern of the application role with the other application roles in the distributed application;

mapping the role profile to a virtual machine (VM) executing the application role;

generating a small state machine from the role profile;

running the small state machine on a port associated with the VM;

inspecting an application level traffic at the port; and

determining an anomaly in traffic of the VM based on a change of state of the small state machine, wherein the anomaly indicates a security breach of the VM.

2. The method of claim 1 , wherein the role profile includes interaction libraries capturing the interaction pattern of the application role.

3. The method of claim 1 , wherein the detecting the application role comprises:

obtaining network traces of the distributed application; and

analyzing the network traces to extract the application role.

4. The method of claim 1 , wherein detection of the security breach comprises:

generating an access control policy for the VM from the role profile; and

determining an anomaly in traffic of the VM based on the access control policy.

5. The method of claim 4 , further comprising:

configuring a firewall with the access control policy.

6. The method of claim 1 , further comprising:

including the role profile in a port profile of the VM.

7. The method of claim 1 , further comprising:

determining a connection graph among the application roles in the distributed application; and

applying the connection graph to an access control list for the VM.

8. The method of claim 1 , further comprising performing a selected one of a group of actions, the group consisting of:

a) logging traffic patterns;

b) re-directing traffic;

c) dropping traffic; and

d) allowing traffic in the network environment.

9. The method of claim 1 , further comprising:

continuously learning the distributed application, which includes adding newly learnt legitimate interactions to the interaction pattern.

10. Logic encoded in non-transitory media that includes instructions for execution and when executed by a processor, is operable to perform operations comprising:

detecting an application role in a plurality of application roles of a distributed application executing in a plurality of virtual machines in a network environment;

generating a role profile for the application role from an interaction pattern of the application role with the other application roles in the distributed application;

mapping the role profile to a virtual machine (VM) executing the application role;

generating a small state machine from the role profile;

running the small state machine on a port associated with the VM;

inspecting an application level traffic at the port; and

determining an anomaly in traffic of the VM based on a change of state of the small state machine, wherein the anomaly indicates a security breach of the VM.

11. The logic of claim 10 , wherein the detecting the application role comprises:

obtaining network traces of the distributed application; and

analyzing the network traces to extract the application role.

12. The logic of claim 10 , wherein detection of the security breach comprises:

generating an access control policy for the VM from the role profile; and

determining an anomaly in traffic of the VM based on the access control policy.

13. The logic of claim 10 , wherein the operations further comprise:

including the role profile in a port profile of the VM.

14. The logic of claim 10 , further comprising:

determining a connection graph among the application roles in the distributed application; and

applying the connection graph to an access control list for the VM.

15. An apparatus, comprising:

a memory element for storing data; and

a processor operable to execute instructions associated with the data, wherein the processor and the memory element cooperate, such that the apparatus is configured for:

detecting an application role in a plurality of application roles of a distributed application executing in a plurality of virtual machines in a network environment;

generating a role profile for the application role from an interaction pattern of the application role with the other application roles in the distributed application;

mapping the role profile to a virtual machine (VM) executing the application role;

generating a small state machine from the role profile;

running the small state machine on a port associated with the VM;

inspecting an application level traffic at the port; and

determining an anomaly in traffic of the VM based on a change of state of the small state machine, wherein the anomaly indicates a security breach of the VM.

16. The apparatus of claim 15 , wherein the detecting the application role comprises:

obtaining network traces of the distributed application; and

analyzing the network traces to extract the application role.

17. The apparatus of claim 15 , wherein detection of the security breach comprises:

generating an access control policy for the VM from the role profile; and

determining an anomaly in traffic of the VM based on the access control policy.

18. The apparatus of claim 15 , further configured for:

including the role profile in a port profile of the VM.

19. The apparatus of claim 15 , further comprising:

determining a connection graph among the application roles in the distributed application; and

applying the connection graph to an access control list for the VM.

20. The apparatus of claim 15 , further comprising performing a selected one of a group of actions, the group consisting of:

a) logging traffic patterns;

b) re-directing traffic;

c) dropping traffic; and

d) allowing traffic in the network environment.