Granted Patent B2

US 11,481,291 · App. 17/146,529 · Granted Oct 25, 2022

Inventors: Lior Kamran (Rishon LeZion, IL); Vladimir Shveidel (Pardes-Hana, IL)

Assignee: EMC IP Holding Company LLC

G06F11/201G06F3/067G06F11/1425G06F11/1612

Abstract

Claims (65)

1. An apparatus comprising:

at least one processing device comprising a processor coupled to a memory;

said at least one processing device being configured:

to associate a first storage node with at least a second storage node in a plurality of storage nodes of a distributed storage system, each such storage node comprising a plurality of storage devices;

to establish a storage devices group comprising specified ones of the storage devices of the first and second storage nodes;

to detect an issue with a first communication channel to the first storage node; and

responsive to the detected issue, to communicate with the first storage node via the second storage node, utilizing an alternative communication channel involving one or more storage devices of the storage devices group;

wherein establishing the storage devices group comprising specified ones of the storage devices of the first and second storage nodes comprises initiating performance of a group formation algorithm in each of at least one of the first and second storage nodes to establish the storage devices group; and

wherein each of the first and second storage nodes independently identifies the specified storage devices of the storage devices group.

2. The apparatus of claim 1 wherein said at least one processing device comprises at least one processing core of a multi-core storage node of the distributed storage system.

3. The apparatus of claim 1 wherein associating the first storage node with at least the second storage node comprises pairing the first storage node with the second storage node.

4. The apparatus of claim 1 wherein the storage devices group comprises a designated subset of storage devices within a common disk array enclosure (DAE) shared by the first and second storage nodes.

5. The apparatus of claim 1 wherein detecting an issue with a first communication channel to the first storage node comprises detecting in a system manager implemented on a given one of the plurality of storage nodes that the first storage node has become disconnected from a mesh network that interconnects the plurality of storage nodes such that the system manager can no longer communicate directly with the first storage node through the mesh network using the first communication channel.

6. The apparatus of claim 5 wherein the given storage node that implements the system manager comprises a storage node of the plurality of storage nodes, other than the first and second storage nodes.

7. The apparatus of claim 5 wherein the system manager is configured, responsive to the detected issue, to communicate with the first storage node via the second storage node, utilizing the alternative communication channel involving one or more storage devices of the storage devices group.

8. The apparatus of claim 1 wherein the group formation algorithm is initiated in each of the first and second storage nodes and comprises:

discovering connected storage devices;

ordering the discovered storage devices using one or more specified criteria; and

selecting particular ones of the ordered storage devices as the storage devices of the storage devices group;

wherein each of the first and second storage nodes by performance of the group formation algorithm independently defines the storage devices group.

9. The apparatus of claim 1 wherein the group formation algorithm is reinitiated in each of the first and second storage nodes responsive to one or more designated events impacting at least one of the discovered storage devices.

10. The apparatus of claim 1 wherein first and second areas are designated on each of two or more of the storage devices of the first and second storage nodes, with the first storage node being configured to write communications of the alternative communication channel to the first areas and to read communications of the alternative communication channel from the second areas, and the second storage node being configured to write communications of the alternative communication channel to the second areas and to read communications of the alternative communication channel from the first areas.

11. The apparatus of claim 10 wherein a system manager implemented on a given one of the plurality of storage nodes, other than the first and second storage nodes, communicates with the first storage node via the second storage node utilizing the alternative communication channel at least in part by:

sending a message to the second storage node; and

receiving a response to the message from the second storage node;

wherein the second storage node writes the message from the system manager into the second area of each of at least a subset of the storage devices of the storage devices group in association with a message identifier;

wherein the first storage node obtains the message written by the second storage node by reading the message from the second area of at least one of the storage devices of the storage devices group;

wherein the first storage node responds to the obtained message by writing a response into the first area of each of at least a subset of the storage devices of the storage devices group in association with the message identifier;

wherein the second storage node obtains the response written by the first storage node by reading the response from the first area of at least one of the storage devices of the storage devices group; and

wherein the second storage node sends the obtained response to the system manager.

12. An apparatus comprising:

at least one processing device comprising a processor coupled to a memory;

said at least one processing device being configured:

to associate a first storage node with at least a second storage node in a plurality of storage nodes of a distributed storage system, each such storage node comprising a plurality of storage devices;

to establish a storage devices group comprising specified ones of the storage devices of the first and second storage nodes;

to detect an issue with a first communication channel to the first storage node; and

responsive to the detected issue, to communicate with the first storage node via the second storage node, utilizing an alternative communication channel involving one or more storage devices of the storage devices group;

wherein the storage devices group comprises a quorum drives group having at least a designated number of storage devices determined based at least in part on a particular redundant array of independent disks (RAID) arrangement utilized by the first and second storage nodes.

13. The apparatus of claim 12 wherein:

(a) the RAID arrangement comprises a RAID 5 arrangement providing protection for one storage device failure at a time, and the designated number of storage devices in the quorum drives group comprises at least three storage devices; or

(b) the RAID arrangement comprises a RAID 6 arrangement providing protection for up to two concurrent storage device failures, and the designated number of storage devices in the quorum drives group comprises at least five storage devices.

14. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to associate a first storage node with at least a second storage node in a plurality of storage nodes of a distributed storage system, each such storage node comprising a plurality of storage devices;

to establish a storage devices group comprising specified ones of the storage devices of the first and second storage nodes;

to detect an issue with a first communication channel to the first storage node; and

responsive to the detected issue, to communicate with the first storage node via the second storage node, utilizing an alternative communication channel involving one or more storage devices of the storage devices group;

wherein establishing the storage devices group comprising specified ones of the storage devices of the first and second storage nodes comprises initiating performance of a group formation algorithm in each of at least one of the first and second storage nodes to establish the storage devices group; and

wherein each of the first and second storage nodes independently identifies the specified storage devices of the storage devices group.

15. The computer program product of claim 14 wherein detecting an issue with a first communication channel to the first storage node comprises detecting in a system manager implemented on a given one of the plurality of storage nodes that the first storage node has become disconnected from a mesh network that interconnects the plurality of storage nodes such that the system manager can no longer communicate directly with the first storage node through the mesh network using the first communication channel.

16. The computer program product of claim 14 wherein first and second areas are designated on each of two or more of the storage devices of the first and second storage nodes, with the first storage node being configured to write communications of the alternative communication channel to the first areas and to read communications of the alternative communication channel from the second areas, and the second storage node being configured to write communications of the alternative communication channel to the second areas and to read communications of the alternative communication channel from the first areas.

17. A method comprising:

associating a first storage node with at least a second storage node in a plurality of storage nodes of a distributed storage system, each such storage node comprising a plurality of storage devices;

establishing a storage devices group comprising specified ones of the storage devices of the first and second storage nodes;

detecting an issue with a first communication channel to the first storage node; and

responsive to the detected issue, communicating with the first storage node via the second storage node, utilizing an alternative communication channel involving one or more storage devices of the storage devices group;

wherein establishing the storage devices group comprising specified ones of the storage devices of the first and second storage nodes comprises initiating performance of a group formation algorithm in each of at least one of the first and second storage nodes to establish the storage devices group;

wherein each of the first and second storage nodes independently identifies the specified storage devices of the storage devices group; and

wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

18. The method of claim 17 wherein detecting an issue with a first communication channel to the first storage node comprises detecting in a system manager implemented on a given one of the plurality of storage nodes that the first storage node has become disconnected from a mesh network that interconnects the plurality of storage nodes such that the system manager can no longer communicate directly with the first storage node through the mesh network using the first communication channel.

19. The method of claim 17 wherein first and second areas are designated on each of two or more of the storage devices of the first and second storage nodes, with the first storage node being configured to write communications of the alternative communication channel to the first areas and to read communications of the alternative communication channel from the second areas, and the second storage node being configured to write communications of the alternative communication channel to the second areas and to read communications of the alternative communication channel from the first areas.

20. The method of claim 17 wherein the group formation algorithm initiated in each of the first and second storage nodes comprises:

discovering connected storage devices;

ordering the discovered storage devices using one or more specified criteria; and

selecting particular ones of the ordered storage devices as the storage devices of the storage devices group;

wherein each of the first and second storage nodes by performance of the group formation algorithm independently defines the storage devices group.

Assignments (9)

Continuity (1)

Cited By (2)

US 12,393,366 US 12,572,297