Using a machine learning classifier to assign a data retention priority for network forensics and retrospective detection

1. A method, comprising:

receiving, at a device in a network, traffic data regarding one or more traffic flows in the network;

applying, by the device, a machine learning classifier to the traffic data;

determining, by the device, a reputation score for an endpoint device or user profile associated with the traffic data;

determining, by the device, a data retention priority for the traffic data based in part on an output of the machine learning classifier and the determined reputation score, wherein the output of the machine learning classifier comprises a probability of the traffic data belonging to a particular class; and

storing, by the device, the traffic data for a period of time based on the determined data retention priority for the traffic data.

2. The method as in claim 1 , further comprising:

determining, by the device, whether the received traffic data comprises flow data for a particular one of the one or more traffic flows, intra-flow data for the particular traffic flow, or raw packet data captured from the particular traffic flow; and

storing, by the device, the received traffic data in a storage repository of a particular repository type based on whether the received traffic data comprises flow data, intra-flow data, or raw packet data.

3. The method as in claim 2 , wherein the period of time is further based on the repository type.

4. The method as in claim 1 , wherein applying the machine learning classifier to the traffic data comprises:

applying the classifier to the traffic data on a per-flow basis to identify a suspicious traffic flow, wherein the classifier is based on prior traffic flows flagged as malicious.

5. The method as in claim 1 , wherein the machine learning classifier is an unsupervised classifier configured to evaluate the anomalousness of a traffic flow.

6. The method as in claim 1 , further comprising:

determining, by the device, the data retention priority for the traffic data based in part on whether the traffic data is associated with one or more hosts in a predefined set of hosts.

7. The method as in claim 6 , further comprising:

generating, by the device, the predefined set of hosts based on an analysis of a plurality of traffic flows in the network.

8. The method as in claim 1 , wherein the reputation score is determined based in part on prior network attacks associated with the endpoint device or user profile.

9. An apparatus, comprising:

one or more network interfaces to communicate with a network;

a processor coupled to the network interfaces and configured to execute one or more processes; and

a memory configured to store a process executable by the processor, the process when executed operable to:

receive traffic data regarding one or more traffic flows in the network;

apply a machine learning classifier to the traffic data;

determine a reputation score for an endpoint device or user profile associated with the traffic data,

determine a data retention priority for the traffic data based in part on an output of the machine learning classifier and the determined reputation score, wherein the output of the machine learning classifier comprises a probability of the traffic data belonging to a particular class; and

store the traffic data for a period of time that is a function of the determined data retention priority for the traffic data.

10. The apparatus as in claim 9 , wherein the process when executed is further operable to:

determine whether the received traffic data comprises flow data for a particular one of the one or more traffic flows, intra-flow data for the particular traffic flow, or raw packet data captured from the particular traffic flow; and

store the received traffic data in a storage repository of a particular repository type based on whether the received traffic data comprises flow data, intra-flow data, or raw packet data, wherein the period of time is further based on the repository type.

11. The apparatus as in claim 9 , wherein the apparatus applies the classifier to the traffic data on a per-flow basis to identify a suspicious traffic flow, wherein the classifier is based on prior traffic flows flagged as malicious.

12. The apparatus as in claim 9 , wherein the machine learning classifier is an unsupervised classifier configured to evaluate the anomalousness of a traffic flow.

13. The apparatus as in claim 9 , wherein the process when executed is further operable to:

determine the data retention priority for the traffic data based in part on whether the traffic data is associated with one or more hosts in a predefined set of hosts.

14. The apparatus as in claim 13 , wherein the process when executed is further operable to:

generate the predefined set of hosts based on an analysis of a plurality of traffic flows in the network.

15. The apparatus as in claim 9 , wherein the reputation score is determined based in part on prior network attacks associated with the endpoint device or user profile.

16. The apparatus as in claim 9 , wherein the process when executed is further operable to:

receive the traffic data from a plurality of networking devices in the network configured to capture the traffic data, wherein the networking devices comprise at least one of: a router, a switch, or a firewall.

17. A tangible, non-transitory, computer-readable medium storing program instructions that cause a device in a network to execute a process comprising:

receiving traffic data regarding one or more traffic flows in the network;

applying a machine learning classifier to the traffic data;

determining a reputation score for one or more characteristics of the traffic data;

determining a data retention priority for the traffic data based in part on an output of the machine learning classifier and the reputation score for the one or more characteristics of the traffic data, wherein the output of the machine learning classifier comprises a probability of the traffic data belonging to a particular class; and

storing the traffic data for a period of time that is a function of the determined data retention priority for the traffic data.