Enabling use of non-volatile media-express (NVMe) over a network
Enabling a protocol for efficiently and reliably using the NVME protocol over a network, referred to as NVME over Network, or NVMEoN, may include an NVMEoN exchange layer for handling exchanges between initiating and target nodes on a network, a burst transmission protocol that provides guaranteed delivery without duplicate retransmission, and an exchange status block approach to manage state information about exchanges.
1. A method for enabling use of NVME (Non-Volatile Media-Express) over a network, the method comprising:
(a) communicating an NVME command from a host of a first node over a PCIe connection of the first node, wherein the PCIe connection couples the host of the first node to a proxy NVME controller of the first node;
(b) terminating a PCIe level transaction of the NVME command communicated over the PCIe connection;
(c) sending a first packet that encapsulates the NVME command using a two-layer NVMEoN (NVME over network) protocol from the proxy NVME controller of the first node and over the network and to a remote virtual controller of a second node, wherein the remote virtual controller of the second node is coupled to a host of the second node by a PCIe connection but wherein the NVME command is received from the first node and is executed but does not pass over the PCIe connection to the host of the second node, wherein the NVMEoN protocol includes an exchange layer and an underlying BTP (Burst Transmission Protocol) layer, wherein the first packet includes an Ethernet header portion, a BTP layer portion, and an exchange layer portion, wherein the exchange layer portion of the first packet contains identification information that identifies an exchange between an NVME initiator and an NVME target, wherein the BTP layer portion of the first packet comprises a first BTP request credit control message that requests credit to send an identified packet via a first burst window to the second node;
(d) receiving a second packet onto the proxy NVME controller of the first node from the second node, wherein the second packet includes an Ethernet header portion, a BTP layer portion, and an exchange layer portion, wherein the exchange layer portion of the second packet contains identification information that identifies the exchange, wherein the BTP layer portion of the second packet comprises a BTP grant credit control message, wherein the BTP grant credit control message does not grant credit to send the identified packet via the first burst window;
(e) receiving a third packet onto the proxy NVME controller of the first node from the second node, wherein the third packet comprises a BTP ACK message, wherein the BTP ACK message signifies a close of the first burst window; and
(f) sending a fourth packet from the proxy NVME controller of the first node over the network to the remote virtual controller of the second node, wherein the fourth packet comprises a second BTP request credit control message that requests credit to send the identified packet via a second burst window to the remote virtual controller of the second node, wherein the network is an Ethernet network, and wherein none of the first, second, third and fourth packets has a TCP header.
2. The method of claim 1 , wherein the BTP layer of the NVMEoN protocol uses BTP grant credit control messages to control packet flow between nodes.
3. The method of claim 1 , wherein the exchange layer portion of the first packet further includes an exchange id, wherein the exchange id identifies the exchange, and wherein the proxy NVME controller of the first node maintains state information associated with the exchange.
4. The method of claim 1 , wherein the BTP layer portion of the first packet further includes a first burst id that identifies the first burst window, and wherein the BTP layer portion of the fourth packet further includes a second burst id that identifies the second burst window.
5. The method of claim 1 , wherein the BTP layer of the NVMEoN protocol guarantees delivery of an NVME commands and guarantees delivery of data packets associated with the exchanges of those NVME commands.
6. The method of claim 1 , wherein the BTP layer of the NVMEoN protocol employs a timer and employs BTP packet sequence ids to detect losses by the Ethernet network of BTP packets.