High speed data packet flow processing with offload
An embodiment may involve digital circuitry configured to: (i) receive a plurality of data packets, (ii) calculate, based on content at a pre-determined set of locations within the data packets, respective hash values for each of the data packets, and (iii) store, in a first memory, metadata containing the respective hash values; and a plurality of processors configured to: (i) read, from the first memory, the metadata, (ii) aggregate, based on the respective hash values, the metadata into flow statistics of flows defined by the data packets, and (iii) write, to a second memory, the flow statistics, wherein the flows are subsets of the data packets having common values in each of the pre-determined set of locations.
1 . A system comprising:
digital circuitry configured to: (i) receive a plurality of data packets, (ii) calculate, based on content at a pre-determined set of locations within the data packets, respective hash values for each of the data packets, and (iii) store, in a first memory, metadata containing the respective hash values; and
a plurality of processors configured to: (i) read, from the first memory, the metadata, (ii) aggregate, based on the respective hash values, the metadata into flow statistics of flows defined by the data packets, and (iii) write, to a second memory, the flow statistics, wherein the flows are subsets of the data packets having common values in each of the pre-determined set of locations.
2 . The system of claim 1 , wherein the pre-determined set of locations is defined by respective byte offsets and byte counts for each of the locations.
3 . The system of claim 1 , wherein the digital circuitry is further configured to:
determine, for each of the data packets, respective characteristics; and
include, in the metadata stored in the first memory and associated with the respective hash values, the respective characteristics.
4 . The system of claim 3 , wherein the respective characteristics include byte lengths for each of the data packets, and wherein the flow statistics include a sum of the byte lengths of the subsets of the data packets in each of the flows.
5 . The system of claim 1 , wherein the digital circuitry and the plurality of processors operate in parallel to one another.
6 . The system of claim 1 , wherein the plurality of processors includes a first processor, a second processor, and a third processor, and wherein aggregating the metadata into the flow statistics of the flows defined by the data packets comprises:
aggregating, by the first processor, the metadata of a first chunk of the data packets into first flow statistics of first flows defined by the first chunk of the data packets;
aggregating, by the second processor, the metadata of a second chunk of the data packets into second flow statistics of second flows defined by the second chunk of the data packets; and
aggregating, by the third processor, the first flow statistics and the second flow statistics.
7 . The system of claim 6 , wherein aggregating the first flow statistics and the second flow statistics comprises summing or averaging at least parts of common flows in the first flow statistics and the second flow statistics.
8 . The system of claim 1 , wherein the digital circuitry is further configured to:
calculate, based on second content at a second pre-determined set of locations within the data packets, respective second hash values for each of the data packets, wherein storing the metadata comprises storing the metadata containing the respective second hash values.
9 . The system of claim 8 , wherein the plurality of processors are further configured to:
aggregate, based on the respective second hash values, the metadata into second flow statistics of second flows defined by the data packets; and
write, to the second memory, the second flow statistics, wherein the second flows are second subsets of the data packets having second common values in each of the second pre-determined set of locations.
10 . The system of claim 1 , wherein the digital circuitry and the plurality of processors are disposed within at least two different devices.
11 . The system of claim 1 , wherein the first memory comprises one or more solid state drives (SSDs) and the second memory comprises a database.
12 . The system of claim 1 , wherein the digital circuitry comprises a field-programmable gate array (FPGA).
13 . A plurality of processors within a computing system, the plurality of processors configured to:
read, from a first memory, metadata containing respective hash values that were calculated based on content at a pre-determined set of locations within a plurality of data packets;
aggregate, based on the respective hash values, the metadata into flow statistics of flows defined by the data packets; and
write, to a second memory, the flow statistics, wherein the flows are subsets of the data packets having common values in each of the pre-determined set of locations.
14 . The plurality of processors of claim 13 , wherein the respective hash values were calculated by digital circuitry that received the plurality of data packets and stored the metadata in the first memory.
15 . The plurality of processors of claim 14 , wherein the digital circuitry further:
determined, for each of the data packets, respective characteristics; and
included, in the metadata stored in the first memory and associated with the respective hash values, the respective characteristics.
16 . The plurality of processors of claim 15 , wherein the respective characteristics include byte lengths for each of the data packets, and wherein the flow statistics include a sum of the byte lengths of the subsets of the data packets in each of the flows.
17 . The plurality of processors of claim 14 , wherein the digital circuitry further calculated, based on second content at a second pre-determined set of locations within the data packets, respective second hash values for each of the data packets, wherein storing the metadata comprises storing the metadata containing the respective second hash values, and wherein the plurality of processors are further configured to:
aggregate, based on the respective second hash values, the metadata into second flow statistics of second flows defined by the data packets; and
write, to the second memory, the second flow statistics, wherein the second flows are second subsets of the data packets having second common values in each of the second pre-determined set of locations.
18 . A method comprising:
reading, from a first memory by a plurality of processors, metadata containing respective hash values that were calculated based on content at a pre-determined set of locations within a plurality of data packets;
aggregating, by the plurality of processors and based on the respective hash values, the metadata into flow statistics of flows defined by the data packets; and
writing, to a second memory by the plurality of processors, the flow statistics, wherein the flows are subsets of the data packets having common values in each of the pre-determined set of locations.
19 . The method of claim 18 , wherein the respective hash values were calculated by digital circuitry that received the plurality of data packets and stored the metadata in the first memory.
20 . The method of claim 19 , wherein the digital circuitry further calculated, based on second content at a second pre-determined set of locations within the data packets, respective second hash values for each of the data packets, wherein storing the metadata comprises storing the metadata containing the respective second hash values, the method further comprising:
aggregating, by the plurality of processors and based on the respective second hash values, the metadata into second flow statistics of second flows defined by the data packets; and
writing, to the second memory by the plurality of processors, the second flow statistics wherein the second flows are second subsets of the data packets having second common values in each of the second pre-determined set of locations.