Network application security policy generation
Embodiments of the present invention generate network communication policies by applying machine learning to existing network communications, and without using information that labels such communications as healthy or unhealthy. The resulting policies may be used to validate communication between applications (or services) over a network.
1. A method performed by at least one computer processor executing computer program instructions stored in at least one non-transitory computer-readable medium, the method comprising:
(A) for each of a plurality of communications over a network between applications executing on a plurality of computer systems, collecting and storing data about the plurality of communications, including, for each of the plurality of communications:
(1) data representing a local Internet Protocol (IP) address, local port, and protocol of the communication;
(2) data representing a remote IP address and remote port of the communication;
(3) data, other than the local IP address, local port, and protocol of the communication, representing a source application of the communication; and
(4) data, other than the remote IP address and remote port of the communication, representing a destination application of the communication;
(B) generating flow data based on the data about the plurality of communications collected and stored in (A), wherein the flow data includes a plurality of flow objects, wherein each of the plurality of flow objects contains data representing communications involving a single corresponding application;
(C) producing match data containing a plurality of match objects, wherein each of the match objects represents a pair of flow objects, in the plurality of flow objects, representing a flow at a source end of a network communication and a flow at a destination end of the network communication wherein the plurality of match objects do not include labels labeling communications as healthy or unhealthy; and
(D) generating a network communication model based on the match data, the network communication model comprising a plurality of rules, each of which comprises at least one feature-value pair representing a network communication from the plurality of communications and a corresponding probability calculated as the occurrence of the network communication containing the at least one feature-value pair, wherein the at least one feature-value pair includes at least one feature from a set comprising a source host, a source application, a destination host, and a destination application.
2. The method of claim 1 , wherein (A) comprises:
(A) (1) at a first local network information collection agent at a first one of the plurality of computer systems, collecting and storing data about a plurality of communications involving the first one of the plurality of computer systems; and
(A) (2) at a second local network information collection agent at a second one of the plurality of computer systems, collecting and storing data about a plurality of communications involving the second one of the plurality of computer systems.
3. The method of claim 2 , wherein (A) further comprises:
(A) (3) at the first local network information collection agent, transmitting the data about the plurality of communications involving the first one of the plurality of computer systems to a remote server; and
(A) (4) at the second local network information collection agent, transmitting the data about the plurality of communications involving the second one of the plurality of computer systems to the remote server.
4. The method of claim 1 , wherein (C) comprises identifying a first flow object containing:
data representing a communication having a particular IP address as its local IP address and a particular port as its local port; and
data representing a communication having the particular IP address as its remote IP address and the particular port as its remote port.
5. The method of claim 1 , wherein (D) comprises generating the network communication model using a MapReduce algorithm.
6. The method of claim 1 , wherein (D) comprises generating the network communication model using an unsupervised decision tree.
7. The method of claim 1 , wherein (D) comprises generating the network communication model using frequent itemset discovery.
8. The method of claim 1 , wherein (D) comprises generating the network communication model using a greedy algorithm.
9. The method of claim 1 , wherein (D) comprises generating the network communication model using a stochastic optimization model.
10. A system comprising at least one non-transitory computer-readable medium containing instructions executable by at least one computer processor to perform a method, the method comprising:
(A) for each of a plurality of communications over a network between applications executing on a plurality of computer systems, collecting and storing data about the plurality of communications, including, for each of the plurality of communications:
(1) data representing a local Internet Protocol (IP) address, local port, and protocol of the communication;
(2) data representing a remote IP address and remote port of the communication;
(3) data, other than the local IP address, local port, and protocol of the communication, representing a source application of the communication; and
(4) data, other than the remote IP address and remote port of the communication, representing a destination application of the communication;
(B) generating flow data based on the data about the plurality of communications collected and stored in (A), wherein the flow data includes a plurality of flow objects, wherein each of the plurality of flow objects contains data representing communications involving a single corresponding application;
(C) producing match data containing a plurality of match objects, wherein each of the match objects represents a pair of flow objects, in the plurality of flow objects, representing a flow at a source end of a network communication and a flow at a destination end of the network communication wherein the plurality of match objects do not include labels labeling communications as healthy or unhealthy; and
(D) generating a network communication model based on the match data, the network communication model comprising a plurality of rules, each of which comprises at least one feature-value pair representing a network communication from the plurality of communications and a corresponding probability calculated as the occurrence of the network communication containing the at least one feature-value pair, wherein the at least one feature-value pair includes at least one feature from a set comprising a source host, a source application, a destination host, and a destination application.
11. The system of claim 10 , wherein (A) comprises:
(A) (1) at a first local network information collection agent at a first one of the plurality of computer systems, collecting and storing data about a plurality of communications involving the first one of the plurality of computer systems; and
(A) (2) at a second local network information collection agent at a second one of the plurality of computer systems, collecting and storing data about a plurality of communications involving the second one of the plurality of computer systems.
12. The system of claim 11 , wherein (A) further comprises:
(A) (3) at the first local network information collection agent, transmitting the data about the plurality of communications involving the first one of the plurality of computer systems to a remote server; and
(A) (4) at the second local network information collection agent, transmitting the data about the plurality of communications involving the second one of the plurality of computer systems to the remote server.
13. The system of claim 10 , wherein (C) comprises identifying a first flow object containing:
data representing a communication having a particular IP address as its local IP address and a particular port as its local port; and
data representing a communication having the particular IP address as its remote IP address and the particular port as its remote port.
14. The system of claim 10 , wherein (D) comprises generating the network communication model using a MapReduce algorithm.
15. The system of claim 10 , wherein (D) comprises generating the network communication model using an unsupervised decision tree.
16. The system of claim 10 , wherein (D) comprises generating the network communication model using frequent itemset discovery.
17. The system of claim 10 , wherein (D) comprises generating the network communication model using a greedy algorithm.
18. The system of claim 10 , wherein (D) comprises generating the network communication model using a stochastic optimization model.