Data flow graph-driven analytics platform using data processing units having hardware accelerators
A data flow graph-driven analytics platform is described in which highly-programmable data stream processing devices, referred to generally herein as data processing units (DPUs), operate to provide a scalable, fast and efficient analytics processing architecture. In general, the DPUs are specialized data-centric processors architected for efficiently applying data manipulation operations (e.g., regular expression operations to match patterns, filtering operations, data retrieval, compression/decompression and encryption/decryption) to streams of data units, such as packet flows having network packets, a set of storage packets being retrieved from or written to storage or other data units.
1. A device comprising:
an analytics interface to receive a request specifying at least one analytical operation to be performed on data;
a query compiler to generate, based on the analytical operation, a data flow graph for configuring at least one data processing unit (DPU) to execute the analytical operation, wherein the DPU comprises an integrated circuit having hardware-based accelerators configured for processing the data as one or more streams of data units, and wherein the data flow graph comprises a data structure having one or more graph nodes connected by one or more directional arcs, each arc representing a stream of data units to be processed or produced by the DPU, and each of the graph nodes represents a set of data stream processing operations to be performed by the DPU to process or produce the data streams; and
a query execution controller configured to communicate the data flow graph to the DPU to configure the DPU to perform the analytical operation on the data.
2. The device of claim 1 , wherein the query execution controller is configured to:
receive one or more data streams that comprise results from application of the analytical operation to the data by the DPU,
construct a response that aggregates the results, and
output the response to an analytics tool that issued the request.
3. The device of claim 1 ,
wherein the query execution controller is configured to select a plurality of the DPUs to perform the analytical operation, and
wherein the query compiler is configured to generate the data flow graph as a plurality of data structures representing a portion of the data flow graph, each portion of the data flow graph configuring a respective one of the DPUs to apply the stream processing operations for the graph nodes specified in that portion of the data flow graph.
4. The device of claim 3 , wherein the query execution controller is configured to communicate each portion of the data flow graph to a corresponding one of the DPUs, each portion of the DPU graph configuring the respective one of the DPUs to apply the stream processing operations for the graph nodes specified in that portion of the data flow graph.
5. The device of claim 1 ,
wherein the request comprises a query specifying at least one query predicate, and
wherein the query compiler is configured to generate the data flow graph to include at least one graph node for configuring the DPU to apply a query predicate to the data by applying a regular expression to one or more of the data streams using a hardware-based regular expression processing accelerator of the DPU.
6. The device of claim 1 , wherein the query compiler is configured to generate the data flow graph to include at least one graph node for configuring the DPU to extract the data from a data source.
7. The device of claim 1 , wherein the query compiler is configured to generate the data flow graph to include at least one graph node for configuring the DPU to output one or more data streams comprising results from application of the analytical operation to the data by the DPU.
8. The device of claim 1 , wherein the query compiler is configured to generate the data flow graph to include at least one graph node for configuring the DPU to apply a function to two or more input streams of data to produce an output stream of data containing aggregate values for values represented in the input streams.
9. The device of claim 1 , wherein the query compiler is configured to generate the data flow graph to include at least one graph node for configuring the DPU to generate a data flow having a sequence of data values starting at an initial value and ending at a final value using a defined increment.
10. The device of claim 1 , wherein the query compiler is configured to generate the data flow graph to include at least one graph node for configuring the DPU to operate on an input data flow to produce an output data flow that sends a maximum number of the data units of the input data flow.
11. The device of claim 1 , wherein the query compiler is configured to generate the data flow graph to include at least one graph node for configuring the DPU to receive an input data flow and invoke one or more of the hardware-based accelerators of the DPU to perform encryption, compression or encoding operations on the input data flow to produce a resultant data flow.
12. The device of claim 11 , wherein the at least one graph node further configures the DPU to store the resultant data flow within storage.
13. A method comprising:
receiving, with a control node, a request specifying at least one analytical operation to be performed on data;
selecting at least one data processing unit (DPU) having hardware-based accelerators configured for processing the data as one or more streams of data units;
generating, based on the analytical operation, a data flow graph for configuring the DPU to execute the analytical operation, wherein generating the data flow graph comprises generating the data flow graph as a data structure having one or more graph nodes connected by directional arcs, each arc representing a stream of data units to be processed or produced by the DPU, and each of the graph nodes represents a set of data stream processing operations to be performed by the DPU to process or produce the data streams; and
communicating the data flow graph to the DPU to configure the DPU to perform the analytical operation on the data.
14. The method of claim 13 further comprising:
receiving, with the control node, one or more data streams from the DPU, wherein the one or more data streams comprise results from application of the analytical operation to the data by the DPU;
constructing a response that aggregates the results; and
outputting, with the control node, the response to an analytics tool that issued the request.
15. The method of claim 13 ,
wherein selecting at least one DPU comprises selecting a plurality of DPUs; and
wherein communicating the data flow graph comprises communicating a respective portion of the data flow graph to each of the DPUs, each portion of the data flow graph configuring a respective one of the DPUs to apply the stream processing operations for the graph nodes specified in that portion of the data flow graph.
16. The method of claim 13 ,
wherein the request comprises a query specifying at least one query predicate, and
wherein generating the data flow graph comprises generating the data flow graph to include at least one graph node for configuring the DPU to apply the query predicate to the data by applying a regular expression to one or more of the data streams using a hardware-based regular expression processing accelerator of the DPU.
17. The method of claim 13 , wherein generating the data flow graph comprises generating the data flow graph to include at least one graph node for configuring the DPU to extract the data from a data source.
18. The method of claim 13 , wherein generating the data flow graph comprises generating the data flow graph to include at least one graph node for configuring the DPU to output one or more data streams comprising results from application of the analytical operation to the data by the DPU.
19. The method of claim 13 , wherein generating the data flow graph comprises generating the data flow graph to include at least one graph node for configuring the DPU to apply a function to two or more input streams of data to produce an output stream of data containing aggregate values for values represented in the input streams.
20. The method of claim 13 , wherein generating the data flow graph comprises generating the data flow graph to include at least one graph node for configuring the DPU to generate a data flow having a sequence of data values starting at an initial value and ending at a final value using a defined increment.
21. The method of claim 13 , wherein generating the data flow graph comprises generating the data flow graph to include at least one graph node for configuring the DPU to operate on an input data flow to produce an output data flow that sends a maximum number of the data units of the input data flow.
22. The method of claim 13 , wherein generating the data flow graph comprises generating the data flow graph to include at least one graph node for configuring the DPU to receive an input data flow and invoke one or more of the hardware-based accelerators of the DPU to perform encryption, compression or encoding operations on the input data flow to produce a resultant data flow.
23. The method of claim 22 , wherein the at least one graph node further configures the DPU to store the resultant data flow within storage.
24. A device comprising:
a data processing unit (DPU) comprising an integrated circuit having programmable processor cores and hardware-based accelerators configured for processing streams of data units; and
a controller to receive a data flow graph and configure the DPU to apply an analytical operation, wherein the data flow graph has one or more graph nodes connected by one or more directional arcs, each arc representing a stream of data units to be processed or produced by the DPU, and each of the graph nodes represents a set of data stream processing operations to be performed by the DPU to process the data streams to apply the analytical operation.
25. The device of claim 24 ,
wherein the hardware-based accelerators include a regular expression engine, and
wherein the data flow graph includes at least one graph node for configuring the regular expression engine of the DPU to apply a query predicate to data by applying a regular expression to one or more of the data streams.
26. The device of claim 24 , wherein, based on graph nodes of the data flow graph, the controller configures the DPU to perform one or more of the following:
extract the data from a data source;
output one or more data streams comprising results from application of the analytical operation;
apply a function to two or more input streams of data to produce an output stream of data containing aggregate values for values represented in the input streams;
generate a data flow having a sequence of data values starting at an initial value and ending at a final value using a defined increment; and
operate on an input data flow to produce an output data flow that sends a maximum number of the data units of the input data flow.
27. The device of claim 26 , wherein, based on graph nodes of the data flow graph, the controller configures the DPU to receive an input data flow and invoke one or more of the hardware-based accelerators of the DPU to perform encryption, compression or encoding operations on the input data flow to produce a resultant data flow.
28. The device of claim 27 , wherein the controller configures the DPU to store the resultant data flow to storage.
29. A system comprising:
a data source storing data;
a plurality of data processing units (DPUs), each DPU having hardware-based accelerators configured for processing streams of data units;
a control node comprising:
an analytics interface to receive a request specifying at least one analytical operation to be performed on the data;
a query compiler to generate a data flow graph for configuring the DPUs to execute the analytical operation, wherein the data flow graph comprises a data structure having one or more graph nodes connected by one or more directional arcs, each arc representing a stream of data units to be processed or produced by one of the DPUs, and each of the graph nodes represents a set of data stream processing operations to be performed by one of the DPUs to process or produce the data streams; and
a query execution controller configured to communicate the data flow graph to the DPUs to configure the DPUs to perform the analytical operation on the data.
30. The system of claim 29 , wherein the control node comprises one of a computing device, a server, a virtual machine, a container or a network appliance.
31. The system of claim 29 , wherein the control node comprises one of a cluster of control nodes associated with a plurality of data processing units for a network.