Automated estimation of network security policy risk
A computer system automatically tests a network communication model by predicting whether particular traffic (whether actual or simulated) should be allowed on the network, and then estimating the accuracy of the network communication model based on the prediction. Such an estimate may be generated even before the model has been applied to traffic on the network. For example, steps can include observing positive data associated with a network; generating a network communication model based on the positive data; generating negative data based on the network communication model; calculating a precision of the network communication model based on the network communication model and the negative data; and calculating an accuracy of the network communication model based on one or more of the precision of the network communication model, or the network communication model and the positive data.
1. A non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors to perform steps of:
collecting and storing positive data associated with real observed communications over a network;
generating a network communication model based on the positive data;
generating negative data based on the network communication model, the negative data representing traffic that the network communication model should not allow, wherein the negative data is generated by collecting all unique pairs from the observed positive data, and generating a plurality of new unique pairs which do not exist in the network communication model, and wherein a unique pair represents a connection between a source host and application and a destination host and application;
calculating a precision of the network communication model based on the network communication model and the negative data; and
calculating an accuracy of the network communication model based on one or more of the precision of the network communication model, or the network communication model and the positive data.
2. The non-transitory computer-readable medium of claim 1 , wherein the negative data is generated to represent negative data that could be observed on the network.
3. The non-transitory computer-readable medium of claim 1 , wherein the negative data is generated based on all of the received positive data, or a portion of the received positive data.
4. The non-transitory computer-readable medium of claim 1 , wherein the network communication model is generated based on positive flow match data representing some or all of observed communications data observed on the network, and wherein observed communications data includes, for each of the plurality of observed communications: data representing a source application of the observed communication, data representing a destination application of the observed communication, data representing a local Internet Protocol (IP) address of the observed communication, and data representing a remote IP address of the observed communication.
5. The non-transitory computer-readable medium of claim 1 , wherein an initial set of negative data is generated, and wherein a subset of the initial negative data is used for the calculating.
6. The non-transitory computer-readable medium of claim 1 , wherein the instructions further cause the one or more processors to perform steps of:
calculating a value of recall for the network communication model; and
calculating the accuracy of the network communication model based on the recall, or based on a combination of the precision and the recall.
7. A method comprising steps of:
collecting and storing positive data associated with real observed communications over a network;
generating a network communication model based on the positive data;
generating negative data based on the network communication model, the negative data representing traffic that the network communication model should not allow, wherein the negative data is generated by collecting all unique pairs from the observed positive data, and generating a plurality of new unique pairs which do not exist in the network communication model, and wherein a unique pair represents a connection between a source host and application and a destination host and application;
calculating a precision of the network communication model based on the network communication model and the negative data; and
calculating an accuracy of the network communication model based on one or more of the precision of the network communication model, or the network communication model and the positive data.
8. The method of claim 7 , wherein the negative data is generated to represent negative data that could be observed on the network.
9. The method of claim 7 , wherein the negative data is generated based on all of the received positive data, or a portion of the received positive data.
10. The method of claim 7 , wherein the network communication model is generated based on positive flow match data representing some or all of observed communications data observed on the network, and wherein observed communications data includes, for each of the plurality of observed communications: data representing a source application of the observed communication, data representing a destination application of the observed communication, data representing a local Internet Protocol (IP) address of the observed communication, and data representing a remote IP address of the observed communication.
11. The method of claim 7 , wherein an initial set of negative data is generated, and wherein a subset of the initial negative data is used for the calculating.
12. The method of claim 7 , further comprising the steps of:
calculating a value of recall for the network communication model; and
calculating the accuracy of the network communication model based on the recall, or based on a combination of the precision and the recall.
13. A system comprising:
one or more processors; and
memory storing computer-executable instructions that, when executed, cause the one or more processors to:
collect and store positive data associated with real observed communications over a network;
generate a network communication model based on the positive data;
generate negative data based on the network communication model, the negative data representing traffic that the network communication model should not allow, wherein the negative data is generated by collecting all unique pairs from the observed positive data, and generating a plurality of new unique pairs which do not exist in the network communication model, and wherein a unique pair represents a connection between a source host and application and a destination host and application;
calculate a precision of the network communication model based on the network communication model and the negative data; and
calculate an accuracy of the network communication model based on one or more of the precision of the network communication model, or the network communication model and the positive data.
14. The system of claim 13 , wherein the negative data is generated to represent negative data that could be observed on the network.
15. The system of claim 13 , wherein the negative data is generated based on all of the received positive data, or a portion of the received positive data.
16. The system of claim 13 , wherein the network communication model is generated based on positive flow match data representing some or all of observed communications data observed on the network, and wherein observed communications data includes, for each of the plurality of observed communications: data representing a source application of the observed communication, data representing a destination application of the observed communication, data representing a local Internet Protocol (IP) address of the observed communication, and data representing a remote IP address of the observed communication.
17. The system of claim 13 , further comprising the steps of:
calculating a value of recall for the network communication model; and
calculating the accuracy of the network communication model based on the recall, or based on a combination of the precision and the recall.