Data processing method and apparatus, distributed data flow programming framework, and related component
A data processing method, a data processing apparatus, a distributed data flow programming framework, an electronic device, and a storage medium. The data processing method includes: dividing a data processing task into a plurality of data processing subtasks (S 101 ); determining, in a Field Programmable Gate Array (FPGA) accelerator side, a target FPGA acceleration board corresponding to each of the data processing subtasks (S 102 ); and sending data to be computed to the target FPGA acceleration board, and executing the corresponding data processing subtask by use of each of the target FPGA acceleration boards to obtain a data processing result (S 103 ). According to the method, a physical limitation of host interfaces on the number of FPGA acceleration boards in an FPGA accelerator side may be avoided, thereby improving the data processing efficiency.
1. A data processing method, for applying to a host side, wherein the host side is connected to a Field Programmable Gate Array (FPGA) accelerator side, the FPGA accelerator side comprises a plurality of FPGA acceleration boards, and the plurality of FPGA acceleration boards are connected to each other through a network, the data processing method comprises:
dividing, by the host side, a data processing task into a plurality of data processing subtasks, and determining target FPGA acceleration boards in the FPGA accelerator side corresponding to each of the plurality of data processing subtasks;
selecting a first target FPGA acceleration board and a second target FPGA acceleration board from all of the target FPGA acceleration boards;
sending all of the plurality of data processing subtasks and task distribution information to the first target FPGA acceleration board, and sending all of data to be computed and data distribution information to the second target FPGA acceleration board: wherein the task distribution information comprises correspondences between the plurality of data processing subtasks and the target FPGA acceleration boards, and the data distribution information comprises correspondences between the data to be computed and the target FPGA acceleration boards;
sending, by the first target FPGA acceleration board, the plurality of data processing subtasks to corresponding target FPGA acceleration boards according to the task distribution information, and sending, by the second target FPGA acceleration board, the data to be computed to corresponding FPGA acceleration boards according to the data distribution information; and
executing a corresponding data processing subtask of the plurality of data processing subtasks by each of the target FPGA acceleration boards to obtain a data processing result, wherein the data processing result comprises intermediate computing data or a final processing result of the data processing task, source data of each of the plurality of data processing subtasks comprises at least one of the data to be computed or the intermediate computing data, and the intermediate computing data is transmitted through the network between the target FPGA acceleration boards.
2. The data processing method of claim 1 , wherein after the sending, by the second target FPGA acceleration board, the data to be computed to corresponding FPGA acceleration boards according to the data distribution information, the method further comprises:
configuring a data address and data length of the data to be computed into the target FPGA acceleration boards, whereby the target FPGA acceleration boards store the data to be computed according to the data address and the data length.
3. The data processing method of claim 1 , further comprising:
configuring a computation result storage address into the target FPGA acceleration boards, whereby the target FPGA acceleration boards store at least one of the intermediate computing data or the final processing result into the computation result storage address.
4. The data processing method of claim 1 , wherein the sending, by the second target FPGA acceleration board, the data to be computed to corresponding FPGA acceleration boards according to the data distribution information comprises:
sending corresponding data to be computed to the target FPGA acceleration boards in a self-defined Remote Direct Memory Access (RDMA) transmission mode; and
correspondingly, the method further comprises:
sending the corresponding data processing subtask to the target FPGA acceleration boards in a preset transmission mode; wherein the preset transmission mode comprises streaming or integral packet transmission.
5. The data processing method of claim 1 , further comprising:
when the target FPGA acceleration boards send the intermediate computing data to other target FPGA acceleration boards based on the network, sending a data address and data length of the intermediate computing data to the other target FPGA acceleration boards, whereby the other target FPGA acceleration boards store the intermediate computing data according to the data address and the data length of the intermediate computing data.
6. The data processing method of claim 1 , wherein data is transmitted between the target FPGA acceleration boards by a Medium Access Control (MAC) interface; data is transmitted between the host side and the target FPGA acceleration boards by a Peripheral Component Interconnect Express (PCIE) interface; and data is transmitted between a remote device and the target FPGA acceleration boards by a second MAC interface.
7. The data processing method of claim 1 , wherein when the host side and the target FPGA acceleration boards are devices in a local area network, a packing operation is performed in a Medium Access Control (MAC) layer on communication data between the host side and the target FPGA acceleration boards; and
when the host side and the target FPGA acceleration boards are devices in a public network, a second packing operation is performed in a User Datagram Protocol (UDP) payload layer on the communication data between the host side and the target FPGA acceleration boards, and a third packing operation is performed in the UDP payload layer on communication data between the target FPGA acceleration boards.
8. A non-transitory computer readable medium comprising computer readable program code that when executed by a processor enables the processor to execute a distributed data flow programming framework, comprising a Central Processing Unit (CPU) acceleration stack and a Field Programmable Gate Array (FPGA) acceleration stack;
the CPU acceleration stack is configured to provide an underlying support for dividing a data processing task, and is further configured to provide a host-side Application Programming Interface (API) for scheduling a data processing subtask;
the FPGA acceleration stack is configured to provide an acceleration interface corresponding to the host-side API, and is further configured to provide an acceleration data control program and a kernel control program during execution of the data processing subtask;
wherein when the distributed data flow programming framework is operating, the CPU acceleration stack divides the data processing task into a plurality of data processing subtasks, and determines target FPGA acceleration boards in an FPGA accelerator side corresponding to each of the plurality of data processing subtasks; selects a first target FPGA acceleration board and a second target FPGA acceleration board from all of the target FPGA acceleration boards; sends all of the plurality of data processing subtasks and task distribution information to the first target FPGA acceleration board, and sends all of data to be computed and data distribution information to the second target FPGA acceleration board; wherein the task distribution information comprises correspondences between the plurality of data processing subtasks and the target FPGA acceleration boards, and the data distribution information comprises correspondences between the data to be computed and the target FPGA acceleration boards; the first target FPGA acceleration board sends the plurality of data processing subtasks to corresponding target FPGA acceleration boards according to the task distribution information, and the second target FPGA acceleration board sends the data to be computed to corresponding FPGA acceleration boards according to the data distribution information; and the FPGA acceleration stack executes a corresponding data processing subtask of the plurality of data processing subtasks by use of each of the target FPGA acceleration boards to obtain a data processing result, wherein the data processing result comprises intermediate computing data or a final processing result of the data processing task, source data of each of the plurality of data processing subtasks comprises at least one of the data to be computed or the intermediate computing data, and the intermediate computing data is transmitted through a network between the target FPGA acceleration boards.
9. The non-transitory computer readable medium of claim 8 , wherein the FPGA acceleration stack comprises:
a static region, configured to execute the plurality of data processing subtasks distributed by the CPU acceleration stack, and further configured to send the data processing result to the CPU acceleration stack or another target FPGA acceleration board; and
an FPGA acceleration engine, configured to send the plurality of data processing subtasks to a local acceleration unit or an acceleration unit of other target FPGA acceleration boards to perform an accelerated computation operation.
10. An electronic device, comprising a memory and a processor, wherein the memory has a computer program thereon; and the processor, when calling the computer program in the memory, implements steps of a data processing method, wherein the data processing method comprises:
dividing, by a host side, a data processing task into a plurality of data processing subtasks, and determining target Field Programmable Gate Array (FPGA) acceleration boards in an FPGA accelerator side corresponding to each of the plurality of data processing subtasks;
selecting a first target FPGA acceleration board and a second target FPGA acceleration board from all of the target FPGA acceleration boards;
sending all of the plurality of data processing subtasks and task distribution information to the first target FPGA acceleration board, and sending all of data to be computed and data distribution information to the second target FPGA acceleration board; wherein the task distribution information comprises correspondences between the plurality of data processing subtasks and the target FPGA acceleration boards, and the data distribution information comprises correspondences between the data to be computed and the target FPGA acceleration boards;
sending, by the first target FPGA acceleration board, the plurality of data processing subtasks to corresponding target FPGA acceleration boards according to the task distribution information, and sending, by the second target FPGA acceleration board, the data to be computed to corresponding FPGA acceleration boards according to the data distribution information; and
executing a corresponding data processing subtask of the plurality of data processing subtasks by each of the target FPGA acceleration boards to obtain a data processing result, wherein the data processing result comprises intermediate computing data or a final processing result of the data processing task, source data of each of the plurality of data processing subtasks comprises at least one of the data to be computed or the intermediate computing data, and the intermediate computing data is transmitted through a network between the target FPGA acceleration boards.
11. The electronic device of claim 10 , wherein after the sending, by the second target FPGA acceleration board, the data to be computed to corresponding FPGA acceleration boards according to the data distribution information, the method further comprises:
configuring a data address and data length of the data to be computed into the target FPGA acceleration boards, whereby the target FPGA acceleration boards store the data to be computed according to the data address and the data length.
12. The electronic device of claim 10 , wherein the data processing method further comprises:
configuring a computation result storage address into the target FPGA acceleration boards, whereby the target FPGA acceleration boards store at least one of the intermediate computing data or the final processing result into the computation result storage address.
13. The electronic device of claim 10 , wherein the sending, by the second target FPGA acceleration board, the data to be computed to corresponding FPGA acceleration boards according to the data distribution information comprises:
sending the corresponding data to be computed to the target FPGA acceleration boards in a self-defined Remote Direct Memory Access (RDMA) transmission mode; and
correspondingly, the method further comprises:
sending the corresponding data processing subtask to the target FPGA acceleration boards in a preset transmission mode; wherein the preset transmission mode comprises streaming or integral packet transmission.
14. The electronic device of claim 10 , wherein the data processing method further comprises:
when the target FPGA acceleration boards send the intermediate computing data to other target FPGA acceleration boards based on the network, sending a data address and data length of the intermediate computing data to the other target FPGA acceleration boards, whereby the other target FPGA acceleration boards store the intermediate computing data according to the data address and the data length of the intermediate computing data.
15. The electronic device of claim 10 , wherein data is transmitted between the target FPGA acceleration boards by a Medium Access Control (MAC) interface; data is transmitted between the host side and the target FPGA acceleration boards by a Peripheral Component Interconnect Express (PCIE) interface; and data is transmitted between a remote device and the target FPGA acceleration boards by a second MAC interface.
16. The electronic device of claim 10 , wherein when the host side and the target FPGA acceleration boards are devices in a local area network, a packing operation is performed in a Medium Access Control (MAC) layer on communication data between the host side and the target FPGA acceleration boards; and
when the host side and the target FPGA acceleration boards are devices in a public network, a second packing operation is performed in a User Datagram Protocol (UDP) payload layer on the communication data between the host side and the target FPGA acceleration boards, and a third packing operation is performed in the UDP payload layer on communication data between the target FPGA acceleration boards.