High-volume network threat trace engine
An approach for high-volume network threat tracing and detection may be implemented by storing network communications received from a plurality of hosts in an initial recording data structure, such as a rolling buffer. Identifiers may be generated for the plurality of hosts associated with the network communications by according to host identity or the behavior of a given host. Extended trace time values may be assigned to a portion of the plurality of hosts based at least in part on the identifiers, and storing the portion of the network communications that have extended trace time values may be recorded as packet capture files in long term memory.
1. A method for selectively storing network traffic, comprising:
storing a set of network packets received from a plurality of hosts in an initial recording data structure, wherein the set of network packets are maintained in the initial recording data structure while the set of network packets are within a corresponding initial trace time window;
storing a subset of the set of network packets in a second recording data structure based at least in part on an identification of anomalous activity of a host, the subset of the set of network packets comprising network packets received from the host of the plurality of hosts, the anomalous activity being identified by processing the subset of the network packets, wherein the host is assigned an extended trace time window and the subset of the set of network packets are maintained in the second recording data structure while the extended trace time window has not expired, wherein the extended trace time window expires later than the initial trace time window; and
storing a new network packet from the host in the second recording data structure based at least in part on a determination that the extended trace time window has not expired.
2. The method of claim 1 , wherein an expiration of an extended trace time window for the host automatically terminates storing network packets generated by the host to the second recording data structure.
3. The method of claim 1 , further comprising storing the new network packet from the host to a long-term memory structure configured to record network traffic perpetually.
4. The method of claim 1 , wherein the extended trace time window are assigned using a lookup memory structure that stores host identifiers and corresponding extended trace time windows to be assigned.
5. The method of claim 1 , wherein the extended trace time window is proportional to how anomalous the anomalous activity is as characterized by a machine learning scheme.
6. The method of claim 1 , wherein the initial trace time window is decreased in response to an amount of the initial recording data structure memory capacity used.
7. The method of claim 1 , wherein the subset of the set of network packets are stored as one or more packet capture files.
8. The method of claim 1 , wherein a portion of the set of network packets are not associated with extended trace time windows, and wherein the portion of the set of network packets is not stored in the second recording data structure.
9. A system for selectively storing network traffic, comprising:
a memory storing a set of program code instructions; and
a processor that executes the set of program code instructions to perform a set of acts, the set of acts comprising:
storing a set of network packets received from a plurality of hosts in an initial recording data structure, wherein the set of network packets are maintained in the initial recording data structure while the set of network packets are within a corresponding initial trace time window;
storing a subset of the set of network packets in a second recording data structure based at least in part on an identification of anomalous activity of a host, the subset of the set of network packets comprising network packets received from the host of the plurality of hosts, the anomalous activity being identified by processing the subset of the network packets, wherein the host is assigned an extended trace time window and the subset of the set of network packets are maintained in the second recording data structure while the extended trace time window has not expired, wherein the extended trace time window expires later than the initial trace time window; and
storing a new network packet from the host in the second recording data structure based at least in part on a determination that the extended trace time window has not expired.
10. The system of claim 9 , wherein an expiration of an extended trace time window for the host automatically terminates storing network packets generated by the host to the second recording data structure.
11. The system of claim 9 , the set of acts further comprising storing the new network packet from the host to a long-term memory structure configured to record network traffic perpetually.
12. The system of claim 9 , wherein the extended trace time window are assigned using a lookup memory structure that stores host identifiers and corresponding extended trace time windows to be assigned.
13. The system of claim 9 , wherein the extended trace time window is proportional to how anomalous the anomalous activity is as characterized by a machine learning scheme.
14. The system of claim 9 , wherein the initial trace time window is decreased in response to an amount of the initial recording data structure memory capacity used.
15. The system of claim 9 , wherein the subset of the set of network packets are stored as one or more packet capture files.
16. A computer-program product embodied on a non-transitory computer readable storage medium having stored thereon a sequence of instructions which, when executed by a processor, causes a set of acts for selectively storing network traffic, the set of acts comprising:
storing a set of network packets received from a plurality of hosts in an initial recording data structure, wherein the set of network packets are maintained in the initial recording data structure while the set of network packets are within a corresponding initial trace time window;
storing a subset of the set of network packets in a second recording data structure based at least in part on an identification of anomalous activity of a host, the subset of the set of network packets comprising network packets received from the host of the plurality of hosts, the anomalous activity being identified by processing the subset of the network packets, wherein the host is assigned an extended trace time window and the subset of the set of network packets are maintained in the second recording data structure while the extended trace time window has not expired, wherein the extended trace time window expires later than the initial trace time window; and
storing a new network packet from the host in the second recording data structure based at least in part on a determination that the extended trace time window has not expired.
17. The computer-program product of claim 16 , wherein an expiration of an extended trace time window for the host automatically terminates storing network packets generated by the host to the second recording data structure.
18. The computer-program product of claim 16 , the set of acts further comprising storing the new network packet from the host to a long-term memory structure configured to record network traffic perpetually.
19. The computer-program product of claim 16 , wherein the extended trace time window are assigned using a lookup memory structure that stores host identifiers and corresponding extended trace time windows to be assigned.
20. The computer-program product of claim 16 , wherein the extended trace time window is proportional to how anomalous the anomalous activity is as characterized by a machine learning scheme.
21. The computer-program product of claim 16 , wherein the initial trace time window is decreased in response to an amount of the initial recording data structure memory capacity used.
22. The computer-program product of claim 16 , wherein the subset of the set of network packets are stored as one or more packet capture files.
23. The computer-program product of claim 16 , wherein a portion of the set of network packets are not associated with extended trace time windows, and wherein the portion of the set of network packets is not stored in the second recording data structure.