End-to-end data plane offloading for distributed storage using protocol hardware and PISA devices
Methods and apparatus for end-to-end data plane offloading for distributed storage using protocol hardware and Protocol Independent Switch Architecture (PISA) devices. Hardware-based data plane forwarding is implemented in compute and storage switches that comprise smart server switches running software executing in a kernel and user space. The compute switch is coupled to one or more compute servers/nodes and the storage server is coupled to one or more storage servers or storage arrays. The hardware-based data plane forwarding facilitates an end-to-end data plane between the computer server(s) and storage server(s)/array(s) that is offloaded to hardware. In one example the software comprises Ceph components used to implement control plane operations in connection with hardware offloaded data plane operations, and storage traffic employs the NVMe-oF protocol and the kernels include NVMe-oF modules. In one aspect the hardware-based data plane forwarding is implemented using programmable P4switch chips. In one aspect the storage and server switches are Top of Rack (ToR) switches.
1. A method implemented in a distributed environment including one or more compute servers in a first rack and one or more storage servers or storage arrays in a second rack, comprising:
receiving, at a first switch, a storage access request originating from a compute server to access a logical storage unit residing on a target storage device in a storage server or storage array;
determining, at the first switch, the target storage device; and
forwarding the storage access request via a forwarding path traversing the first switch to the target storage device;
wherein:
the first switch comprises a processor to execute software that is to receive write data from an application associated with the compute server;
execution of the software by the processor is to result in distributed replicating of the data in the one or more storage servers or storage arrays; and
the first switch also comprises programmable hardware to detect failure of the target storage device and to notify the software of the failure.
2. The method of claim 1 , wherein the forwarding path is implemented in a data plane offloaded to hardware.
3. The method of claim 2 , wherein the compute server is connected to the first switch via a second link, the first switch is connected to a second switch via a second link, and the second switch is connected to the storage server or storage array via a third link, and wherein the forwarding path in the data plane offloaded to hardware traverses the first link, the first switch, the second link, the second switch, and the third link.
4. The method of claim 1 , wherein the first switch comprises a server switch including hardware comprising a switch chip and one or more processors coupled to memory in which software components are executed, the switch chip programmed to implement hardware-based data plane operations.
5. The method of claim 4 , wherein the switch chip comprises a programmable switch chip that is programmed using the P4 programming language.
6. The method of claim 4 , wherein the software components include software modules configured to implement control plane operations in connection with the offloaded hardware data plane.
7. The method of claim 4 , wherein:
the first switch comprises a Ceph RBD (Reliable Autonomic Distributed Object Store (RADOS) Block Device) module; and
the Ceph RBD module is to be executed on the processor in the first switch.
8. The method of claim 1 , wherein the data transfers between the compute server and the storage server or storage array are facilitated using an NVMe-oF (Non-volatile Memory Express over Fabric) protocol.
9. The method of claim 8 , wherein the data transfers between the compute server and the storage server or storage array are facilitated using a Remote Direct Memory Access (RDMA) protocol in combination with the NVMe-oF protocol.
10. The method of claim 1 , wherein the logical storage unit comprises one of a storage object, storage block or file.
11. A system, comprising:
a compute server, installed in a first rack, including memory connected to one or more processors on which software components are executed including an operating system and software executing in a user space including a first virtual compute host;
a storage server or storage array, including a plurality of storage devices, installed in a second rack;
a compute Top of Rack (ToR) switch installed in the first rack, communicatively coupled to the compute server and including one or more processors on which software instructions are executed for implementing control plane operations and including hardware configured to support data plane forwarding operations; and
a storage Top of Rack (ToR) switch installed in the second rack, communicatively coupled with the compute ToR switch and the storage server or storage array, including one or more processors on which software instructions are executed for implementing control plane operations and including hardware configured to support data plane forwarding operations,
wherein the system is configured to enable the first virtual compute host to access one or more of the plurality of storage devices in the storage server or storage array using an end-to-end data plane offloaded to hardware; and
also wherein:
the one or more processors of the compute ToR switch are to execute other software that is to receive write data from an application associated with the compute server;
execution of the other software by the one or more processors of the compute ToR switch is to result in distributed replicating of the data in the storage server or storage array; and
the compute ToR switch also comprises programmable hardware to detect failure of a target storage device and to notify the other software of the failure.
12. The system of claim 11 , wherein each of the compute ToR switch and the storage ToR switch comprises a server switch including hardware comprising a hardware-programmable switch chip programmed to implement hardware-based data plane forwarding operations.
13. The system of claim 11 , wherein the software components are configured to implement control plane operations in connection with the offloaded hardware data plane operations to support end-to-end data transfers between the first virtual compute host and the one or more of the plurality of storage devices.
14. The system of claim 13 , wherein:
the compute ToR switch includes and executes, at least in part, a Ceph RBD (Reliable Autonomic Distributed Object Store (RADOS) Block Device) module; and
the storage ToR switch includes and executes, at least in part, a Ceph Object Storage Daemon (OSD) module.
15. The system of claim 11 , wherein the data transfers between the first virtual compute host and the one or more of the plurality of storage devices employ a NVMe-oF (Non-volatile Memory Express over Fabric) protocol.
16. A server switch, comprising:
a network interface controller (NIC);
a first central processing unit (CPU);
memory coupled to the first CPU, having an address space logically partitioned to include a kernel space and a user space; and
a programmable switch chip, operatively coupled to the first CPU, the memory, and the NIC,
wherein the programmable switch chip is programmed using a hardware programming language to implement hardware-based data plane operations under which packets associated with storage traffic originating from or destined to virtual compute hosts running on one or more compute servers are forwarded via hardware-based data plane operations implemented in the programmable switch chip when the one or more compute servers are communicatively coupled to the NIC either directly or via a second server switch; and
also wherein:
the programmable switch chip is to execute software that is to receive write data from an application associated with the one or more compute servers;
execution of the software by the programmable switch chip is to result in distributed replicating of the data in a storage server or storage array; and
the programmable switch chip comprises hardware to detect failure of a target storage device and to notify the software of the failure.
17. The server switch of claim 16 , further comprising user space software implementing control plane operations that are performed in connection with forwarding the storage traffic originating from the virtual compute hosts running on the one or more compute servers.
18. The server switch of claim 16 , further comprising a compute Top of Rack (ToR) switch configured to be communicatively coupled to the one or more compute servers, and also further comprising Ceph RBD (Reliable Autonomic Distributed Object Store (RADOS) Block Device) module configured to be executed in the user space of the address space of the memory coupled to the first CPU.
19. The server switch of claim 16 , wherein the second server switch comprises a storage Top of Rack (ToR) switch configured to be communicatively coupled to the one or more storage servers or to one or more storage arrays, and also further comprising a Ceph Object Storage Daemon (OSD) module configured to be executed by the second server switch.
20. The server switch of claim 16 , wherein the storage traffic comprising Non-Volatile Memory Express over Fabric (NVMe-oF) traffic.