Enabling use of non-volatile media—express (NVME) over a network
Enabling a protocol for efficiently and reliably using the NVME protocol over a network, referred to as NVME over Network, or NVMEoN, may include an NVMEoN exchange layer for handling exchanges between initiating and target nodes on a network, a burst transmission protocol that provides guaranteed delivery without duplicate retransmission, and an exchange status block approach to manage state information about exchanges.
1. A method for enabling use of NVME (Non-Volatile Media—Express) over a network, the method comprising:
(a) receiving a first packet from a proxy NVME controller of a first node onto a remote virtual NVME controller of a third node, wherein the proxy NVME controller of the first node is a first sender that communicates with the remote virtual NVME controller of the third node using an NVME over Network (NVMEoN) protocol, wherein the NVMEoN protocol has an exchange layer and an underlying (BTP) Burst Transmission Protocol layer, wherein the first packet includes an Ethernet header portion, a BTP layer portion, and an exchange layer portion, wherein the exchange layer portion of the first packet contains identification information that identifies a first exchange between an NVME initiator and an NVME target, wherein the BTP layer portion of the first packet comprises a BTP request credit control message from the first sender to a receiver;
(b) receiving a second packet from a proxy NVME controller of a second node onto the remote virtual NVME controller of the third node, wherein the proxy NVME controller of the second node is a second sender that communicates with the remote virtual NVME controller of the third node using the NVME over Network (NVMEoN) protocol, wherein the second packet includes an Ethernet header portion, a BTP layer portion, and an exchange layer portion, wherein the exchange layer portion of the second packet contains identification information that identifies a second exchange between an NVME initiator and an NVME target, wherein the BTP layer portion of the second packet comprises a BTP request credit control message from the second sender to a receiver;
(c) sending a third packet from the remote virtual NVME controller to the proxy NVME controller of the first node, wherein the third packet includes an Ethernet header portion, a BTP layer portion, and an exchange layer portion, wherein the BTP layer portion is a BTP grant credit control message, wherein the BTP grant credit control message of the third packet is sent in (c) in response to the BTP request credit control message of the first packet; and
(d) sending a fourth packet from the remote virtual NVME controller to the proxy NVME controller of the second node, wherein the fourth packet includes an Ethernet header portion, a BTP layer portion, and an exchange layer portion, wherein the BTP layer portion is a BTP grant credit control message, wherein the BTP grant credit control message of the fourth packet is sent in (d) in response to the BTP request credit control message of the second packet, wherein the remote virtual NVME controller grants credits in (c) and (d) on a sender-specific basis such that packet flow is controlled differently for different senders in accordance with Quality of Service (QoS) policies.
2. The method of claim 1 , wherein the BTP layer portion of the first packet includes a plurality of BTP packet sequence ids, wherein the BTP layer portion of the second packet includes a plurality of BTP packet sequence ids, wherein the BTP layer portion of the third packet includes a plurality of BTP packet sequence ids, and wherein the BTP portion of the fourth packet includes a plurality of BTP packet sequence ids.
3. The method of claim 2 , wherein the BTP grant credit control message of the third packet grants credits for some but not all of the sequence ids of the BTP layer portion of the first packet.
4. The method of claim 3 , wherein the first node in response to receiving the BTP grant credit control message of the third packet sends a fifth packet to the remote virtual NVME controller of the third node requesting credit for sequence ids that were listed in the BTP request credit control message of the first packet but for which credit was not granted by the BTP grant credit control message of the third packet.
5. The method of claim 1 , wherein the BTP layer portion of the first packet further includes a BTP channel id, and wherein the remote virtual NVME controller of the third node maintains state information associated with a BTP channel identified by the BTP channel id.
6. The method of claim 1 , wherein the exchange layer portion of the first packet further includes an exchange id, wherein the exchange id identifies the first exchange, and wherein the remote virtual NVME controller of the third node maintains state information associated with the first exchange.
7. The method of claim 1 , wherein the remote virtual NVME controller of the third node maintains a BTP layer state machine and an exchange layer state machine.
8. The method of claim 1 , wherein the first packet encapsulates a first NVME command, and wherein the second packet encapsulates a second NVME command.
9. The method of claim 1 , wherein the network is an Ethernet network, and wherein none of the first, second, third and fourth packets has a TCP header.
10. The method of claim 1 , wherein the first node causes a burst window to be set up for communicating packets from the first node to the third node, wherein the proxy NVME controller of the first node controls packet flow from the first node to the third node by requesting credit for more or fewer identified packets to be communicated via the burst window.
11. The method of claim 1 , wherein the first node causes a burst window to be set up for communicating packets from the first node to the third node, wherein the remote virtual NVME controller of the third node controls packet flow from the first node to the third node by controlling the packets for which the remote virtual NVME controller of the third node grants credit to be communicated via the burst window.
12. The method of claim 1 , wherein the network is an Ethernet network, and wherein the remote virtual NVME controller of the third node employs a timer and employs BTP packet sequence ids to detect losses by the Ethernet network of BTP packets.
13. The method of claim 1 , wherein the third node further comprises a host, wherein the host of the third node is coupled to the remote virtual NVME controller of the third node via a PCIe (Peripheral Component Interconnect Express) bus, wherein a BTP packet encapsulates an NVME I/O command, wherein the BTP packet passes from the first node and over the network and into the third node via a network interface port of the remote virtual NVME controller such that the NVME I/O command is executed without the NVME I/O command ever passing over the PCIe bus of the third node.
14. A system comprising:
a first node, wherein the first node comprises a host and a proxy NVME (Non-Volatile Media—Express) controller, wherein the proxy NVME controller is coupled to the host via a host interface bus, wherein a plurality of first NVME commands are passed from the host of the first node, across the host interface bus of the first node, to the proxy NVME controller of the first node, and out of the first node over a network;
a second node, wherein the second node comprises a host and a proxy NVME (Non-Volatile Media—Express) controller, wherein the proxy NVME controller is coupled to the host via a host interface bus, wherein a plurality of second NVME commands are passed from the host of the second node, across the host interface bus of the second node, to the proxy NVME controller of the second node, and out of the second node over the network;
non-volatile storage; and
a third node, wherein the third node comprises a host and a remote virtual NVME controller, wherein the remote virtual NVME controller is coupled to the non-volatile storage so that the remote virtual NVME controller can access data stored in the non-volatile storage, wherein the proxy NVME controller of the first node is a first sender that communicates the plurality of first NVME commands to the remote virtual NVME controller of the third node using a NVME over Network (NVMEoN) protocol, wherein the proxy NVME controller of the second node is a second sender that communicates the plurality of second NVME commands to the remote virtual NVME controller of the third node using the NVMEoN protocol, wherein the NVMEoN protocol has an exchange layer and an underlying (BTP) Burst Transmission Protocol layer, wherein the proxy NVME controller of the first node sends the remote virtual NVME controller of the third node a first BTP request credit control message such that the remote virtual NVME controller of the third node returns a first BTP grant credit for current burst window control message back to the first node, and wherein the proxy NVME controller of the second node sends the remote virtual NVME controller of the third node a second BTP request credit control message such that the remote virtual NVME controller of the third node returns a second BTP grant credit for current burst window control message back to the second node, and wherein the remote virtual NVME controller grants credits to the proxy NVME controller of the first node and to the proxy NVME controller of the second node on a sender-specific basis such that packet flow is controlled differently for different senders in accordance with Quality of Service (QoS) policies.