Enforcing transaction order in peer-to-peer interactions
A method for computing includes submitting a first command from a central processing unit (CPU) to a first peripheral device in a computer to write data in a first bus transaction over a peripheral component bus in the computer to a second peripheral device in the computer. A second command is submitted from the CPU to one of the first and second peripheral devices to execute a second bus transaction, subsequent to the first bus transaction, that will flush the data from the peripheral component bus to the second peripheral device. The first and second bus transactions are executed in response to the first and second commands. Following completion of the second bus transaction, the second peripheral device processes the written data in.
1. A method for computing, comprising:
receiving in a network interface controller (NIC), which is configured as a first peripheral device in a computer and couples the computer to a packet data network, data transmitted over the network in a remote direct memory access (RDMA) operation;
writing the data in a first bus transaction over a peripheral component bus in the computer from the NIC to a second peripheral device in the computer;
after executing the first bus transaction, submitting a completion notification from the NIC to a central processing unit (CPU);
in response to the completion notification, submitting a command from the CPU to one of the first and second peripheral devices to execute a second bus transaction, subsequent to the first bus transaction, that will flush the data from the peripheral component bus to the second peripheral device;
executing the second bus transaction in response to the command; and
following completion of the second bus transaction, processing the written data in the second peripheral device.
2. The method according to claim 1 , wherein submitting the command comprises instructing the first peripheral device to transmit an instruction over the bus to the second peripheral device to process the written data, and wherein executing the second bus transaction both flushes the data and causes the second peripheral device to execute the instruction and process the written data.
3. The method according to claim 1 , wherein submitting the command comprises instructing the second peripheral device to execute the second bus transaction and, after completion of the second bus transaction, to process the written data.
4. The method according to claim 1 , wherein submitting the command comprises instructing the first peripheral device to execute the second bus transaction, and wherein the method comprises, after executing the second bus transaction, submitting a completion notification from the first peripheral device to the CPU, wherein the CPU instructs the second peripheral device to process the written data in response to the completion notification.
5. The method according to claim 1 , wherein executing the first bus transaction comprises writing the data to a memory of the second peripheral device in a posted write operation by direct memory access (DMA) over the bus.
6. The method according to claim 1 , wherein executing the second bus transaction comprises executing, by the one of the first and second peripheral devices to which the command was submitted, a read transaction directed over the bus to the other of the first and second peripheral devices.
7. The method according to claim 1 , wherein receiving the data comprises receiving one or more RDMA packets over the network from a server, wherein the RDMA operation is initiated by a host processor in the server.
8. The method according to claim 1 , wherein submitting the command comprises instructing the NIC to transmit a data packet to a network address that is associated with the second peripheral device.
9. The method according to claim 1 , wherein the second peripheral device comprises a solid state drive (SSD).
10. A method for computing, comprising:
submitting a first command from a central processing unit (CPU) to a first peripheral device in a computer to write data in a first bus transaction over a peripheral component bus in the computer to a second peripheral device in the computer;
submitting a second command from the CPU to one of the first and second peripheral devices to execute a second bus transaction, subsequent to the first bus transaction, that will flush the data from the peripheral component bus to the second peripheral device;
executing the first and second bus transactions in response to the first and second commands; and
following completion of the second bus transaction, processing the written data in the second peripheral device
wherein the first peripheral device is a network interface controller (NIC), which couples the computer to a packet data network, and wherein executing the first bus transaction comprises receiving the data at the NIC over the network in a remote direct memory access (RDMA) operation and writing the received data to the second peripheral device, and
wherein the second peripheral device is a graphics processing unit (GPU).
11. A computing apparatus, comprising:
a network interface controller (NIC), which is configured as a first peripheral device and couples the apparatus to a packet data network;
a second peripheral device;
a peripheral component bus that interconnects at least the first and second peripheral devices; and
a central processing unit (CPU),
wherein the NIC is configured to receive data transmitted over the network in a remote direct memory access (RDMA) operation, to write the data in a first bus transaction over the peripheral component bus from the NIC to the second peripheral device, and after executing the first bus transaction, submitting a completion notification from the NIC to the CPU, and
wherein the CPU is configured, in response to the completion notification, to submit a command to one of the first and second peripheral devices to execute a second bus transaction, subsequent to the first bus transaction, that will flush the data from the peripheral component bus to the second peripheral device, and to cause the second peripheral device to process the written data after execution of the first and second bus transactions has been completed.
12. The apparatus according to claim 11 , wherein the command instructs the first peripheral device to transmit an instruction over the bus to the second peripheral device to process the written data, and wherein executing the second bus transaction both flushes the data and causes the second peripheral device to execute the instruction and process the written data.
13. The apparatus according to claim 11 , wherein the command instructs the second peripheral device to execute the second bus transaction and, after completion of the second bus transaction, to process the written data.
14. The apparatus according to claim 11 , wherein the command instructs the first peripheral device to execute the second bus transaction, and wherein the first peripheral device is configured to submit, after executing the second bus transaction, a completion notification to the CPU, wherein the CPU instructs the second peripheral device to process the written data in response to the completion notification.
15. The apparatus according to claim 11 , wherein the first bus transaction comprises writing the data to a memory of the second peripheral device in a posted write operation by direct memory access (DMA) over the bus.
16. The apparatus according to claim 11 , wherein the second bus transaction comprises a read transaction by the one of the first and second peripheral devices to which the command was submitted, directed over the bus to the other of the first and second peripheral devices.
17. The apparatus according to claim 11 , wherein the NIC is configured to receive the data in one or more RDMA packets over the network from a server, wherein the RDMA operation is initiated by a host processor in the server.
18. The apparatus according to claim 11 , wherein the command instructs the NIC to transmit a data packet to a network address that is associated with the second peripheral device.
19. The apparatus according to claim 11 , wherein the second peripheral device comprises a solid state drive (SSD).
20. A computing apparatus, comprising:
a first peripheral device;
a second peripheral device;
a peripheral component bus that interconnects at least the first and second peripheral devices; and
a central processing unit (CPU), which is configured to a submit a first command to the first peripheral device to write data in a first bus transaction over the peripheral component bus to the second peripheral device, and to submit a second command to one of the first and second peripheral devices to execute a second bus transaction, subsequent to the first bus transaction, that will flush the data from the peripheral component bus to the second peripheral device, and to cause the second peripheral device to process the written data after execution of the first and second bus transactions, in response to the first and second commands, has been completed,
wherein the first peripheral device is a network interface controller (NIC), which couples the computer to a packet data network, and wherein the NIC is configured to receive the data over the network in a remote direct memory access (RDMA) operation and to write the received data to the second peripheral device in the first bus transaction, and
wherein the second peripheral device is a graphics processing unit (GPU).