High speed data packet flow processing
An embodiment may involve a network interface configured to capture data packets into a binary format and a non-volatile memory configured to temporarily store the data packets received by way of the network interface. The embodiment may also involve a first array of processing elements each configured to independently and asynchronously: (i) read a chunk of data packets from the non-volatile memory, (ii) identify flows of data packets within the chunk, and (iii) generate flow representations for the flows. The embodiment may also involve a second array of processing elements configured to: (i) receive the flow representations from the first array of processing elements, (ii) identify and aggregate common flows across the flow representations into an aggregated flow representation, (iii) based on a filter specification, remove one or more of the flows from the aggregated flow representation, and (iv) write information from the aggregated flow representation to the database.
1. A system comprising:
a first plurality of processors configured to perform a first set of operations that involve: (i) reading, by a plurality of conversion applications, chunks of data packets independently and in parallel from memory, (ii) identifying flows of the data packets within the chunks, (iii) generating a plurality of flow representations for the flows, one or more flow representations per conversion application, and (iv) discarding the data packets for which the flows were identified; and
a second plurality of processors configured to perform a second set of operations that involve: (i) receiving the flow representations from the first plurality of processors, (ii) identifying and aggregating common flows across the plurality of flow representations into an aggregated flow representation, and (iii) writing information from the aggregated flow representation to a database.
2. The system of claim 1 , wherein the first set of operations and the second set of operations are performed in parallel.
3. The system of claim 1 , wherein the first set of operations and the second set of operations are performed by at least two different devices.
4. The system of claim 1 , further comprising:
a third plurality of processors configured to execute a protocol decoding application, wherein execution of the protocol decoding application involves: (i) receiving the data packets, (ii) applying a data structure containing a protocol format definition to a payload contained within the data packets, and (iii) based on the applying of the data structure to the payload, generating a representation of sections of the payload in a structured data format.
5. The system of claim 4 , wherein the third plurality of processors overlaps at least in part with the second plurality of processors.
6. The system of claim 4 , wherein execution of the protocol decoding application also involves storing the sections of the payload in an associative array, and wherein generating the representation of the sections of the payload comprises generating the representation of the sections from the associative array.
7. The system of claim 4 , wherein the structured data format is based on JavaScript Object Notation (JSON), and wherein the representation of the sections of the payload are defined without a JSON schema.
8. The system of claim 1 , wherein the plurality of flow representations are in an intermediate format that aggregates header information and metadata associated with the data packets respectively corresponding to the flows, and wherein the metadata includes one or more of a count of the data packets or a count of bytes in the data packets, a device identifier for the system, or a physical port through which the data packets were received by the system.
9. The system of claim 1 , wherein aggregating the common flows across the plurality of flow representations into the aggregated flow representation comprises summing respective packet counts or byte counts from the common flows in the aggregated flow representation.
10. The system of claim 1 , wherein identifying flows of data packets within the chunks comprises calculating, based on header field values of the data packets within the chunks, respective hash values, wherein the respective hash values uniquely denote corresponding flows to which the data packets belong.
11. The system of claim 1 , further comprising:
a third plurality of processors configured to read data packets from a network interface module in hard real-time with latencies within a first threshold.
12. The system of claim 11 , wherein the first set of operations and the second set of operations are performed in soft real-time with average latency within a second threshold, wherein the second threshold is greater than the first threshold.
13. The system of claim 1 , wherein different processors of the second plurality of processors perform operations each of: identifying and aggregating common flows, removing the flows from the aggregated flow representation, and writing the information from the aggregated flow representation to the database.
14. The system of claim 1 , wherein a filter specification passes the flows that match a whitelist or the filter specification passes the flows that are in a set of top m flows in terms of number of data packets or number of bytes, wherein m is between 1 and 10,000.
15. The system of claim 1 , further comprising:
a further array of processors configured to provide a virtual environment, wherein a packet processing application is executable on the virtual environment, and wherein a forwarding buffer allows the packet processing application to read data packets from the memory.
16. A method comprising:
performing, by a first plurality of processors, a first set of operations that involve: (i) reading, by a plurality of conversion applications, chunks of data packets independently and in parallel from memory, (ii) identifying flows of the data packets within the chunks, (iii) generating a plurality of flow representations for the flows, one or more flow representations per conversion application, and (iv) discarding the data packets for which the flows were identified; and
performing, by a second plurality of processors, a second set of operations that involve: (i) receiving the flow representations from the first plurality of processors, (ii) identifying and aggregating common flows across the plurality of flow representations into an aggregated flow representation, and (iii) writing information from the aggregated flow representation to a database.
17. The method of claim 16 , further comprising:
performing, by a third plurality of processors configured to execute a protocol decoding application: (i) receiving the data packets, (ii) applying a data structure containing a protocol format definition to a payload contained within the data packets, and (iii) based on the applying of the data structure to the payload, generating a representation of sections of the payload in a structured data format.
18. The method of claim 17 , wherein the third plurality of processors overlaps at least in part with the second plurality of processors.
19. The method of claim 17 , wherein execution of the protocol decoding application also involves storing the sections of the payload in an associative array, and wherein generating the representation of the sections of the payload comprises generating the representation of the sections from the associative array.
20. The method of claim 17 , wherein the structured data format is based on JavaScript Object Notation (JSON), and wherein the representation of the sections of the payload are defined without a JSON schema.