Distributed raid rebuild
A technique is disclosed for generating rebuild data of a RAID configuration having one or more failed drives. The RAID configuration includes multiple sets of drives coupled to respective computing nodes, and the computing nodes are coupled together via a network. A lead node directs rebuild activities, communicating with the other node or nodes and directing such node(s) to compute partial rebuild results. The partial rebuild results are based on data of the drives of the RAID configuration coupled to the other node(s). The lead node receives the partial rebuild results over the network and computes complete rebuild data based at least in part on the partial rebuild results.
1. A method of generating rebuild data for a RAID (Redundant Array of Independent Disks) configuration having one or more failed drives, comprising:
sending, by a lead node, a message to one or more other computing nodes coupled to the lead node over a network, each of said one or more other computing nodes coupled to a respective set of drives that are part of the RAID configuration, the message directing said one or more other computing nodes to generate partial rebuild results, wherein the lead node is selected as the node coupled to the most drives that belong to the RAID configuration of any node coupled to drives that belong to the RAID configuration;
receiving the partial rebuild results by the lead node; and
generating complete rebuild data for rebuilding said one or more failed drives based at least in part on the partial rebuild results.
2. The method of claim 1 , wherein the lead node is coupled to a set of drives that are part of the RAID configuration, and wherein generating the complete rebuild data is further based on data from the set of drives coupled to the lead node.
3. The method of claim 1 , wherein said one or more other computing nodes include multiple other computing nodes, and wherein directing said one or more other computing nodes to generate partial rebuild results includes directing each of the other computing nodes to generate respective partial rebuild results based on the respective set of drives coupled thereto.
4. The method of claim 3 , wherein said multiple other computing nodes are coupled to the lead node in a chain in which the lead node forms one end of the chain, and wherein the method further comprises:
at least one of the multiple other computing nodes in the chain consolidating its own partial rebuild results with partial rebuild results from at least one more-distal computing node in the chain to produce consolidated partial rebuild results,
wherein receiving the partial rebuild results by the lead node includes receiving the consolidated partial rebuild results.
5. The method of claim 1 , wherein sending the message by the lead node includes transmitting the message via an API (Application Program Interface) between the lead node and said one or more other computing nodes.
6. The method of claim 5 , wherein the API is based on RDMA (Remote Direct Memory Access).
7. The method of claim 5 , wherein sending the message includes providing a list of {drive, offset} pairs to said one or more other computing nodes, the list of {drive, offset} pairs identifying portions of drives that store data used in generating the partial rebuild results.
8. The method of claim 7 , wherein sending the message further includes providing a buffer size, the buffer size specifying an amount of memory to be allocated for holding data used in generating the partial rebuild results.
9. The method of claim 7 , wherein sending the message further includes providing a set of Galois field coefficients used in generating at least some of the partial rebuild results.
10. The method of claim 1 , wherein the partial rebuild results received from at least one of said one or more other computing nodes include results calculated from at least two drives.
11. A computerized apparatus, comprising control circuitry that includes a set of processing units coupled to memory, the control circuitry constructed and arranged to:
send, by a lead node via an API (Application Program Interface), a message to one or more other computing nodes coupled to the lead node over a network, each of said one or more other computing nodes coupled to a respective set of drives that are part of the RAID (Redundant Array of Independent Disks) configuration, the message directing said one or more other computing nodes to generate partial rebuild results;
receive the partial rebuild results by the lead node; and
generate complete rebuild data for rebuilding said one or more failed drives based at least in part on the partial rebuild results,
wherein the control circuitry constructed and arranged to send the message is further constructed and arranged to provide a list of {drive, offset} pairs to said one or more other computing nodes, the list of {drive, offset} pairs identifying portions of drives that store data used in generating the partial rebuild results.
12. The computerized apparatus of claim 11 , wherein the lead node is coupled to a set of drives that are part of the RAID configuration, and wherein the control circuitry constructed and arranged to generate the complete rebuild data is further constructed and arranged to generate the complete rebuild data based on data from the set of drives coupled to the lead node.
13. A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry of a computerized apparatus, cause the computerized apparatus to perform a method of generating rebuild data for a RAID (Redundant Array of Independent Disks) configuration having one or more failed drives, the method comprising:
sending, by a lead node, a message to multiple other computing nodes coupled to the lead node over a network, each of said other computing nodes coupled to a respective set of drives that are part of the RAID configuration, the message directing said other computing nodes to generate partial rebuild results based on the respective sets of drives coupled thereto;
receiving the partial rebuild results by the lead node; and
generating complete rebuild data for rebuilding said one or more failed drives based at least in part on the partial rebuild results,
wherein said multiple other computing nodes are coupled to the lead node in a chain in which the lead node forms one end of the chain, wherein the method further comprises at least one of the multiple other computing nodes in the chain consolidating its own partial rebuild results with partial rebuild results from at least one more-distal computing node in the chain to produce consolidated partial rebuild results, and wherein receiving the partial rebuild results by the lead node includes receiving the consolidated partial rebuild results.
14. The computer program product of claim 13 , wherein the lead node is coupled to a set of drives that are part of the RAID configuration, and wherein generating the complete rebuild data is further based on data from the set of drives coupled to the lead node.
15. The computer program product of claim 13 , wherein sending the message by the lead node includes transmitting the message via an API (Application Program Interface) between the lead node and said one or more other computing nodes, wherein the API is based on RDMA (Remote Direct Memory Access).
16. The computer program product of claim 15 , wherein sending the message includes providing a list of {drive, offset} pairs to said one or more other computing nodes, the list of {drive, offset} pairs identifying portions of drives that store data used in generating the partial rebuild results.
17. The computer program product of claim 16 , wherein sending the message further includes providing a set of Galois field coefficients used in generating at least some of the partial rebuild results.
18. The computer program product of claim 13 , further comprising the selecting the lead node as the node coupled to the most drives that belong to the RAID configuration of any node coupled to drives that belong to the RAID configuration.