System and method for establishing a peer connection using reliable RDMA primitives
A system and method for reliable peer-to-peer communication is provided. The system and method includes a cluster connection manager that acts as a unified resource for opening, closing and maintaining peer-to-peer communication channels between applications executing on storage systems comprising a storage system cluster. The cluster connection manager communicates with cluster connection manager clients to maintain and manage the communication channels. The cluster connection manager may also be operatively interconnected with a plurality of cluster interconnect devices, thereby enabling failover operation in the event that a cluster interconnect device suffers an error condition.
1. A method for initiating a peer-to-peer communication session, the method comprising:
initiating a boot process;
initializing a cluster connection manager in the booting process before a storage operating system executing on a cluster partner is fully active;
initiating, by the cluster connection manger, a first remote direct memory access (RDMA) read operation directed to a cluster partner before a storage operating system executing on the cluster partner is fully active, the RDMA read operation bypassing the operating system;
performing, in response to a successful first RDMA read operation, a first RDMA write operation to the cluster partner;
performing, in response to a successful RDMA write operation, a second RDMA read operation directed to the cluster partner; and
performing, in response to a successful second RDMA read operation, a second RDMA write operation to the cluster partner before a storage operating system executing on the cluster partner is fully active, in the booting process.
2. The method of claim 1 wherein the step of attempting a first RDMA read operation further comprises the step of issuing a RDMA read operation to the cluster partner requesting a pre-set memory address location that is associated with a status variable on the cluster partner.
3. The method of claim 1 further comprising:
exchanging a set of peer connection information;
passing a set of client information to the cluster partner;
creating a set of appropriate communication ports;
alerting the cluster partner of a ready status; and
alerting a set of clients that the cluster partner is in a ready state.
4. The method of claim 3 wherein the set of peer connection information comprises a version number.
5. The method of claim 3 wherein the step of passing a set of client information to the cluster partner further comprises: collecting, from a set of clients, the set of client information; and transferring the collected set of client information to the cluster partner.
6. The method of claim 5 wherein the client information comprises a number of communication ports required.
7. The method of claim 5 wherein the set of client information further comprises an amount of memory requested by a particular client.
8. The method of claim 1 wherein the cluster partner is a storage system.
9. The method of claim 1 wherein the cluster partner is an application server.
10. A method comprising:
initiating a boot process;
initializing a cluster connection manager in the boot process before a storage operating system executing on a cluster partner is fully active;
initiating, a peer-to-peer communication session, by a cluster connection manager, before a storage operating system executing on the cluster partner is fully active which bypasses an operating system on a storage system by attempting a first remote direct memory access read operation directed to a predefined hardware address and a predefined port number, the predefined hardware address and the predefined port number previously known to support a RDMA operation; and
performing, in the booting process before a storage operation system executing on the cluster partner is fully active, in response to a successful initiating, a first remote direct memory access write operation directed to the predefined hardware address and the predefined port number.
11. The method of claim 10 further comprising:
performing, in response to a successful first remote direct memory access write, a second remote direct memory access read operation directed to the predefined hardware address and the predefined port number.
12. The method of claim 10 wherein the predefined hardware address comprises a fibre channel identifier.
13. The method of claim 10 wherein the predefined port number comprises a virtual interface.
14. The method of claim 10 wherein the first remote direct memory access is delivered to a predefined memory address storing booting status information.
15. A system configured to establish reliable peer-to-peer communication among storage system of a clustered environment, the system comprising:
a booting process executed by a processor;
a peer process executing on each storage system partner having an operating system; and
a cluster connection manager executing on each storage system partner, the cluster connection manager establishing a reliable peer-to-peer connection between each peer process in the booting process before a storage operating system executing on a cluster partner is fully active be connecting to a predetermined port number using a predetermined network address, the reliable peer-to-peer connection bypassing the operating system and initiate a remote direct memory access (RDMA) read operation directed to a cluster partner.
16. The system of claim 15 wherein the reliable peer-to-peer connection is established without requiring a storage operating system executing on each storage system partner to be fully functioning.
17. The system of claim 15 wherein the peer-to-peer connection is a virtual interface connection.
18. The system of claim 15 wherein the peer process is a cluster connection client that requests services from the cluster connection manager.
19. A system configured to open initial peer-to-peer connection over a cluster interconnect, the system comprising:
a storage system having an operating system;
a booting process executed by a processor;
a cluster connection manager executing on the storage system, the cluster connection manager configured to establish a peer connection in the booting process before a storage operating system executing on a cluster partner in fully active on a predetermined port number and using a predetermined network address with in the storage system the peer-to-peer connection bypassing the operating system and initiate a remote direct memory access (RDMA) read operation directed to a cluster partner; and
a process executing on the storage system, the process configured to use the established peer connection for communication.
20. The system of claim 19 wherein the peer-to-peer connection is a virtual interface connection.
21. The system of claim 19 wherein the process executing on the storage system is a cluster connection client that requests services from the cluster connection manager.
22. The system of claim 19 wherein the process executing on the storage system communicates with a cluster partner using the established peer connection.
23. A system configured to accept the initiation of a peer-to-peer connection over a cluster interconnect, the system comprising:
a storage system having an operating system;
a booting process executed by a process;
a cluster connection manager executing on the storage system, the cluster connection manager executing on the storage system, the cluster connection manager configured to accept a peer connection on a predetermined port number and using a predetermined network address within the storage system and initiate a remote direct memory access (RDMA) read operation to a cluster partner in the booting process before a storage operating system executing fully active; and
a process executing on the storage system, the process configured to read information from the established peer connection.
24. The system of claim 23 wherein the peer-to-peer connection is a virtual interface connection.
25. The system of claim 23 wherein the process executing on the storage system is a cluster connection client that requests services from the cluster connection manager.
26. The system of claim 23 wherein the process executing on the storage system reads information from a cluster partner.
27. The system of claim 23 wherein the information comprises heartbeat signals.
28. A computer readable medium containing executable program instructions executed by a processor, comprising:
program instructions that initiate a booting process;
program instructions that initialize a cluster connection manager in the booting process before a storage operating system executing a cluster partner is fully active;
program instructions that initiate in the booting process, a first remote direct memory access (RDMA) read operation before a storage operating system executing on a cluster partner is fully active;
program instructions that perform, in response to a successful RDMA write operation, a second RDMA read operation directed to the cluster partner; and
program instructions that perform in the booting process before a storage cooperating system executing on the cluster partner is fully active, in response to a successful second RDMA read operation, a second RDMA write operation to the cluster partner.