Efficient storage architecture for high speed packet capture
An embodiment may involve a network interface module; volatile memory configured to temporarily store data packets received from the network interface module; high-speed non-volatile memory; an interface connecting to low-speed non-volatile memory; a first set of processors configured to perform a first set of operations that involve: (i) reading the data packets from the volatile memory, (ii) arranging the data packets into chunks, each chunk containing a respective plurality of the data packets, and (iii) writing the chunks to the high-speed non-volatile memory; and a second set of processors configured to perform a second set of operations in parallel to the first set of operations, where the second set of operations involve: (i) reading the chunks from the high-speed non-volatile memory, (ii) compressing the chunks, (iii) arranging the chunks into blocks, each block containing a respective plurality of the chunks, and (iv) writing the blocks to the low-speed non-volatile memory.
1. A system comprising:
a network interface module;
volatile memory configured to temporarily store data packets received by way of the network interface module;
high-speed non-volatile memory;
an interface connecting the system to low-speed non-volatile memory, wherein the high-speed non-volatile memory has lower write latency and less storage capacity than the low-speed non-volatile memory;
a first set of processors configured to perform a first set of operations that involve: (i) reading the data packets from the volatile memory, (ii) arranging the data packets into chunks thereof, each chunk containing a respective plurality of the data packets, and (iii) writing the chunks to the high-speed non-volatile memory; and
a second set of processors configured to perform a second set of operations in parallel to the first set of operations performed by the first set of processors, wherein the second set of operations involve: (i) reading the chunks from the high-speed non-volatile memory, (ii) compressing the chunks, (iii) arranging the chunks into blocks thereof, each block containing a respective plurality of the chunks, and (iv) writing, by way of the interface, the blocks to the low-speed non-volatile memory.
2. The system of claim 1 , wherein the interface connects the system to the low-speed non-volatile memory by way of a local-area network.
3. The system of claim 1 , wherein the interface connects the system to the low-speed non-volatile memory by way of a wide-area network.
4. The system of claim 1 , wherein the second set of operations also involves adding error correcting codes to the chunks before writing the blocks to the low-speed non-volatile memory.
5. The system of claim 1 , wherein the second set of operations also involves generating packet-identifying metadata for each of the data packets and storing the metadata separately from the blocks in the low-speed non-volatile memory.
6. The system of claim 5 , wherein the metadata includes copies of fields from layer 2, layer 3, or layer 4 headers of the data packets.
7. The system of claim 6 , wherein the metadata is stored as a histogram of the fields from the layer 2, layer 3, or layer 4 headers of the data packets.
8. The system of claim 1 , wherein the first set of operations is performed in hard real-time with latencies within a first threshold.
9. The system of claim 8 , wherein the second set of operations is performed in soft real-time with average latency within a second threshold, wherein the second threshold is greater than the first threshold.
10. The system of claim 1 , wherein the high-speed non-volatile memory comprises an array of solid state drives (SSDs).
11. The system of claim 1 , wherein the low-speed non-volatile memory comprises an array of hard disk drives (HDDs).
12. The system of claim 1 , wherein non-volatile memory stores indications of whether each of the chunks is stored in the high-speed non-volatile memory or the low-speed non-volatile memory.
13. The system of claim 12 , wherein the first set of operations also involves setting the indications to specify the high-speed non-volatile memory for the chunks written to the high-speed non-volatile memory, and wherein the second set of operations also involves setting the indications to specify the low-speed non-volatile memory for the chunks that are contained within the blocks written to the low-speed non-volatile memory.
14. The system of claim 1 , further comprising:
a volatile memory cache configured to store copies of the chunks from the high-speed non-volatile memory that were recently read by the second set of processors, wherein reading the chunks from the high-speed non-volatile memory comprises:
reading a first subset of the chunks from the volatile memory cache when the volatile memory cache contains the first subset of the chunks; and
reading a second subset of the chunks directly from the high-speed non-volatile memory when the volatile memory cache does not contain the second subset of the chunks.
15. The system of claim 1 , wherein the second set of operations also involve one or more of: (i) converting pluralities of the data packets stored in the chunks to a different format, (ii) generating reverse domain name system entries based on the pluralities of the data packets stored in the chunks, (iii) generating transport-layer security certificates based on the pluralities of the data packets stored in the chunks, (iv) indexing the pluralities of the data packets stored in the chunks, or (v) operating an intrusion detection system based on the pluralities of the data packets stored in the chunks.
16. A method comprising:
performing, by a first set of processors, a first set of operations involving: (i) reading data packets from a volatile memory, wherein the data packets were stored in the volatile memory by a network interface module, (ii) arranging the data packets into chunks thereof, each chunk containing a respective plurality of the data packets, and (iii) writing the chunks to a high-speed non-volatile memory; and
performing, by a second set of processors and in parallel to the first set of operations performed by the first set of processors, a second set of operations involving: (i) reading the chunks from the high-speed non-volatile memory, (ii) compressing the chunks, (iii) arranging the chunks into blocks thereof, each block containing a respective plurality of the chunks, and (iv) writing, by way of an interface, the blocks to a low-speed non-volatile memory, wherein the high-speed non-volatile memory has lower write latency and less storage capacity than the low-speed non-volatile memory.
17. The method of claim 16 , wherein the second set of operations also involves generating packet-identifying metadata for each of the data packets and storing the metadata separately from the blocks in the low-speed non-volatile memory.
18. The method of claim 16 , wherein the first set of operations is performed in hard real-time with latencies within a first threshold.
19. The method of claim 18 , wherein the second set of operations is performed in soft real-time with average latency within a second threshold, wherein the second threshold is greater than the first threshold.
20. The method of claim 16 , wherein non-volatile memory stores indications of whether each of the chunks is stored in the high-speed non-volatile memory or the low-speed non-volatile memory.