Efficient non-transparent bridge (NTB) based data transport
Techniques for providing an efficient NTB-based data transport in a highly available storage system. The techniques include performing an ordered transfer of a source data buffer from a primary storage node to a secondary storage node over a PCIe NTB, writing a “transfer ID value” for the ordered transfer over the NTB to a “received transfer ID register” of the secondary storage node, performing a remote procedure call (RPC) to send, over a side channel, the transfer ID value as a “sent transfer ID value” to the secondary storage node, and processing the RPC call to verify the value contained in the received transfer ID register against the sent transfer ID value. Having performed the verification, the secondary storage node determines a successful or unsuccessful status of the ordered transfer, and sends an RPC reply to acknowledge or inform the primary storage node of the successful or unsuccessful status.
1. A method of providing an efficient non-transparent bridge-based data transport in a highly available storage system, the method comprising:
initiating an ordered transfer of a source data buffer over a non-transparent bridge (NTB) from a first storage node to a second storage node in the highly available storage system;
without waiting for completion of the ordered transfer, writing a transfer identifier (ID) value for the ordered transfer over the NTB from the first storage node to a specified register of the second storage node;
making a remote procedure call (RPC call) at the first storage node to send, over a side channel, the transfer ID value for the ordered transfer to the second storage node;
processing the RPC call at the second storage node to verify the transfer ID value written to the specified register against the sent transfer ID value, thereby determining a status of the ordered transfer; and
having determined the status of the ordered transfer, sending an RPC reply, over the side channel, from the second storage node to acknowledge or inform the primary storage node of the status of the ordered transfer of the source data buffer.
2. The method of claim 1 wherein the transfer ID value is one of a series of transfer ID values written to the specified register of the second storage node, and wherein the writing of the transfer ID value includes overwriting a transfer ID value previously written to the specified register of the second storage node.
3. The method of claim 2 further comprising:
obtaining the transfer ID value for the ordered transfer by incrementing the transfer ID value previously written to the specified register of the second storage node.
4. The method of claim 1 wherein the initiating of the ordered transfer of the source data buffer includes initiating a first PCIe write operation, and wherein the first PCIe write operation has a plurality of parameters including at least a memory address and length of the source data buffer and a destination memory address.
5. The method of claim 4 wherein the memory address and length of the source data buffer corresponds to a first NTB memory window in a first memory of the first storage node, wherein the destination memory address corresponds to a second NTB memory window in a second memory of the second storage node, and wherein the method further comprises:
performing the first PCIe write operation by asynchronously transferring the source data buffer over the NTB from the first NTB memory window to the second NTB memory window.
6. The method of claim 4 wherein the writing of the transfer ID value for the ordered transfer includes initiating a second PCIe write operation, wherein the second PCIe write operation has at least one parameter including the transfer ID value.
7. The method of claim 6 wherein the specified register of the second storage node is mapped to the NTB, and wherein the method further comprises:
performing the second PCIe write operation by asynchronously sending the transfer ID value over the NTB to the specified register of the second storage node.
8. The method of claim 1 wherein the processing of the RPC call at the second storage node includes processing the RPC call after a predefined polling period.
9. The method of claim 1 wherein the processing of the RPC call at the second storage node includes determining whether the transfer ID value contained in the specified register of the second storage node is less than, greater than, or equal to the sent transfer ID value.
10. The method of claim 9 wherein the processing of the RPC call at the second storage node further includes, if the transfer ID value contained in the specified register of the second storage node is less than the sent transfer ID value, performing one or more of (i) entering a holding mode, and (ii) sending an RPC reply over the side channel directing the first storage node to repeat the ordered transfer.
11. The method of claim 9 wherein the processing of the RPC call at the second storage node further includes, if the transfer ID value contained in the specified register of the second storage node is greater than or equal to the sent transfer ID value, determining that the ordered transfer of the source data buffer has been completed.
12. The method of claim 11 further comprising:
sending, over the side channel, an RPC reply to the first storage node to acknowledge completion of the ordered transfer of the source data buffer.
13. The method of claim 1 wherein the RPC call has at least one parameter including the cyclic redundancy code (CRC) of the source data buffer, and wherein the method further comprises:
verifying a CRC of the source data buffer transferred to the second storage node against the CRC of the source data buffer provided as a parameter of the RPC call.
14. A highly available storage system, comprising:
a primary storage processor; and
a secondary storage processor,
wherein the primary storage processor is communicably connected over a non-transparent bridge (NTB) to the secondary storage processor,
wherein the primary storage processor is configured to:
initiate an ordered transfer of a source data buffer over the NTB to the secondary storage processor;
without waiting for completion of the ordered transfer, write a transfer identifier (ID) value for the ordered transfer over the NTB to a specified register of the secondary storage processor; and
make a remote procedure call (RPC call) to send, over a side channel, the transfer ID value for the ordered transfer to the secondary storage processor, and wherein the secondary storage processor is configured to:
process the RPC call to verify the transfer ID value written to the specified register against the sent transfer ID value, thereby determining a status of the ordered transfer; and
having determined the status of the ordered transfer, send an RPC reply, over the side channel, to acknowledge or inform the primary storage processor of the status of the ordered transfer of the source data buffer.
15. The storage system of claim 14 wherein the secondary storage processor is further configured to determine whether the transfer ID value contained in the specified register of the secondary storage node is less than, greater than, or equal to the sent transfer ID value.
16. The storage system of claim 15 wherein the secondary storage processor is further configured, if the transfer ID value contained in the specified register of the secondary storage node is less than the sent transfer ID value, to perform one or more of (i) entering a holding mode, and (ii) sending an RPC reply over the side channel directing the first storage node to repeat the ordered transfer.
17. The storage system of claim 15 wherein the secondary storage processor is further configured, if the transfer ID value contained in the specified register of the secondary storage node is greater than or equal to the sent transfer ID value, to determine that the ordered transfer of the source data buffer has been completed.
18. The storage system of claim 17 wherein the secondary storage processor is further configured to send, over the side channel, an RPC reply to the first storage node to acknowledge completion of the ordered transfer of the source data buffer.
19. A computer program product including a set of non-transitory, computer-readable media having instructions that, when executed by processing circuitry of a first storage node and a second storage node communicably connected to one another over a non-transparent bridge (NTB) in a highly available storage system, cause the processing circuitry to perform a method comprising:
initiating an ordered transfer of a source data buffer over the NTB from the first storage node to the second storage node;
without waiting for completion of the ordered transfer, writing a transfer identifier (ID) value for the ordered transfer over the NTB from the first storage node to a specified register of the second storage node;
making a remote procedure call (RPC call) at the first storage node to send, over a side channel, the transfer ID value for the ordered transfer to the second storage node;
processing the RPC call at the second storage node to verify the transfer ID value written to the specified register against the sent transfer ID value, thereby determining a status of the ordered transfer; and
having determined the status of the ordered transfer, sending an RPC reply, over the side channel, from the second storage node to acknowledge or inform the primary storage node of the status of the ordered transfer of the source data buffer.
20. The computer program product of claim 19 wherein the transfer ID value is one of a series of transfer ID values written to the specified register of the second storage node, wherein the writing of the transfer ID value includes overwriting a transfer ID value previously written to the specified register of the second storage node, and wherein the method further comprises:
obtaining the transfer ID value for the ordered transfer by incrementing the transfer ID value previously written to the specified register of the second storage node.