Control plane method and apparatus for providing erasure code protection across multiple storage devices
A system and method for providing erasure code protection across multiple storage devices. A data switch in a storage system connects a plurality of storage devices to a remote host. Each storage device is also connected to a controller, e.g., a baseboard management controller. During normal operation, read and write commands from the remote host are sent to respective storage devices through the data switch. When a write command is executed, the storage device executing the command sends a copy of the data to the controller, which generates and stores erasure codes, e.g., on a storage device that is dedicated to the storage of erasure codes, and invisible to the remote host. When a device fails or is removed, the controller reconfigures the data switch to redirect all traffic addressed to the failed or absent storage device to the controller, and the controller responds to host commands in its stead.
1. A system comprising:
a plurality of storage devices, comprising first storage devices and second storage devices different from the first storage devices, and configured to store erasure codes sufficient to recover data stored on the first storage devices from data on the second storage devices;
a switch, configured during a first mode of operation to route storage commands received through a storage interface associated with the plurality of storage devices to respective storage devices from the plurality of storage devices, based at least in part on addresses associated with the storage commands; and
a controller,
the controller being configured to, in response to satisfying a first condition, perform operations comprising:
reconfiguring the switch to route storage commands of a first type to the controller; and
sending, in response to receiving the storage commands of the first type, a command response, through at least one of the switch or the storage interface.
2. The system of claim 1 , wherein:
the first condition is satisfied based, at least in part on, a storage device of the first storage devices failing or being absent, and
storage commands of the first type are storage commands associated with an address of the storage device that has failed or become absent.
3. The system of claim 2 , wherein:
storage commands of a second type are storage commands associated with addresses of storage devices other than the storage device that has failed or become absent; and
during the first mode of operation, and after reconfiguring of the switch by the controller in response to satisfying the first condition, the switch is configured to send a command response, generated in response to a storage command of the second type by a storage device other than the storage device that has failed or become absent, through the storage interface.
4. The system of claim 1 , wherein:
one of the first storage devices is configured, during the first mode of operation, upon receipt of a write command including a data block, to store the data block and to send a copy of the data block to the controller; and
the controller is configured, upon receipt of a copy of a data block from a first storage device of the plurality of storage devices, to generate erasure codes sufficient to recover the data block, and to store the erasure codes on one or more of the second storage devices.
5. The system of claim 1 , wherein one of the first storage devices is configured, during the first mode of operation, upon receipt of a write command including a data block:
to store the data block;
to generate erasure codes sufficient to recover the data block; and
to store the erasure codes on one or more of the second storage devices.
6. The system of claim 1 , wherein:
one of the second storage devices is configured to store erasure codes; and
the switch is configured, during the first mode of operation, to route a storage command received through the storage interface to a respective storage device of a subset of the plurality of storage devices, according to a respective address in the storage command, the one of the second storage devices not being a member of the subset.
7. The system of claim 1 , wherein the controller is configured, in the event of failure or absence of any one of the plurality of storage devices, in response to a received read command requesting a data block:
to read erasure codes from the present and correctly operating storage devices of the plurality of storage devices;
to recover the data block; and
to send the recovered data block through the switch and through the storage interface as part of a command response.
8. The system of claim 1 , wherein:
one of the storage devices comprises a connector having a presence pin, the presence pin being connected to a respective sensing pin of the controller; and
the controller is configured to detect removal of the storage device by a change in a voltage or current at the respective sensing pin.
9. The system of claim 1 , wherein the switch is an Ethernet switch.
10. The system of claim 1 , further comprising a PCIe switch, wherein one of the plurality of storage devices is connected to the controller through the PCIe switch.
11. The system of claim 1 , wherein:
the switch has a PCIe control port, connected to a port of the controller, and the switch is configured to receive configuration commands from the controller through the PCIe control port; and
the controller is a baseboard management controller.
12. A method of operating a storage system comprising a plurality of storage devices, a switch, and a controller, the method comprising:
during a first mode of operation, routing storage commands received through a storage interface associated with the plurality of storage devices to respective storage devices from the plurality of storage devices, based at least in part on addresses associated with the storage commands; and
in response to satisfying a first condition, performing operations comprising:
reconfiguring the switch to route storage commands of a first type to the controller; and
sending, in response to receiving the storage commands of the first type, a command response, through at least one of the switch and the storage interface.
13. The method of claim 12 , wherein:
the first condition is satisfied based, at least in part on a storage device of the plurality of storage devices failing or being absent, and
storage commands of the first type are storage commands associated with an address of the storage device that has failed or become absent.
14. The method of claim 13 , wherein:
storage commands of a second type are storage commands associated with addresses of storage devices other than the storage device that has failed or become absent; and
during the first mode of operation, and after reconfiguring of the switch by the controller in response to satisfying the first condition, the switch is configured to send a command response, generated in response to a storage command of the second type by a storage device other than the storage device that has failed or become absent, through the storage interface.
15. The method of claim 12 , further comprising, during the first mode of operation, upon receipt, by a storage device of the plurality of storage devices, of a write command including a data block:
storing, by the storage device, the data block; and
sending, by the storage device, a copy of the data block to the controller.
16. The method of claim 15 , further comprising, upon receipt, by the controller, of a copy of a data block from a first storage device of the plurality of storage devices:
generating, by the controller, erasure codes sufficient to recover the data block; and
storing, by the controller, the erasure codes on one or more storage devices other than the first storage device, from among the plurality of storage devices.
17. The method of claim 12 , further comprising, during the first mode of operation, upon receipt by a first storage device, of the plurality of storage devices, of a write command including a data block:
storing, by the first storage device, the data block;
generating, by the first storage device, erasure codes sufficient to recover the data block; and
storing, by the first storage device, the erasure codes on one or more storage devices other than the first storage device, from among the plurality of storage devices.
18. The method of claim 12 , wherein:
a first storage device of the plurality of storage devices is configured to store erasure codes; and
the method comprises, during the first mode of operation, routing a storage command received through the storage interface to a respective storage device of a subset of the plurality of storage devices, according to a respective address in the storage command, the first storage device not being a member of the subset.
19. The method of claim 12 , wherein the storage system further has a PCIe switch and the controller is connected to the storage devices through the PCIe switch,
the method further comprising, in the event of failure or absence of any one of the plurality of storage devices, in response to receiving, by the controller, of a read command requesting a data block:
reading, by the controller, through the PCIe switch, erasure codes from the present and correctly operating storage devices of the plurality of storage devices;
recovering, by the controller, the data block; and
sending, by the controller, the recovered data block through the switch and through the storage interface as part of a command response.
20. The method of claim 19 , wherein one of the storage devices has a presence pin connected to a respective sensing pin of the controller; and
the method further comprises detecting, by the controller, removal of any one of the storage devices by a change in a voltage or current at the respective sensing pin.
21. A storage system, the storage system comprising:
a plurality of storage devices, comprising first storage devices and second storage devices different from the first storage devices, and configured to store erasure codes sufficient to recover data stored on the first storage devices from data on the second storage devices;
a switch, configured during a first mode of operation to route storage commands received through a storage interface associated with the plurality of storage devices to respective storage devices from the plurality of storage devices, based at least in part on addresses associated with the storage commands; and
a controller, the controller being configured to, in response to satisfying a first condition, perform operations comprising:
reconfiguring the switch to route storage commands of a first type to the controller; and
sending, in response to receiving the storage commands of the first type, a command response, through at least one of the switch and the storage interface.