Concurrent and incremental repair of a failed component in an object based storage system for high availability
View Patent ↗Systems and methods for repairing a failed component within a data storage subsystem without interruption of service. A client device is coupled to a data storage cluster comprising a plurality of data storage locations addressable as rows and columns in an array. Each column of the array comprising a separate computer interconnected to other columns via a network. After both detecting a failure of a first column in the array and detecting a predetermined time threshold has elapsed, a predetermined reconstruction column coordinates reconstructing data storage locations within the reconstruction column. For a second row in the array, a predetermined coordinating column coordinates the execution of a received memory access requests from the client corresponding to the second row. In this manner, a column coming back online after a failure is able to repair without halting the servicing of read and write requests.
1. A computer system comprising:
a client configured to convey read and write requests; and
a data storage cluster coupled to receive the read and write requests, wherein the data storage cluster comprises a plurality of data storage locations addressable as rows and columns in an array;
wherein in response to detecting a failure of a first column, the data storage cluster is configured to:
designate a second column as a replacement for the first column;
initiate reconstruction of first data stored in the first column; and
store reconstructed data corresponding to the first data in the second column;
wherein the data storage cluster is further configured to service received read and write requests during said reconstruction, whereby read data is returned to clients responsive to received read requests and/or write data is written to the array responsive to received write requests; and
wherein during said reconstruction, the data storage cluster is further configured to store information corresponding to a write request targeted to the second column in a temporary log.
2. The system as recited in claim 1 , wherein each column of the array comprises a separate computer interconnected with other columns of the array via a network.
3. The system as recited in claim 1 , wherein during said reconstruction, the data storage cluster is further configured to:
receive a read request targeted to a given row; and
service columns targeted by the read request, except for the second column.
4. The system as recited in claim 3 , wherein the data storage cluster is further configured to designate the second column as a permanent replacement for the first column.
5. The system as recited in claim 4 , wherein the data storage cluster is further configured to designate the second column as the permanent replacement in response to detecting a predetermined amount of time has elapsed after detecting the failure of the first column.
6. The system as recited in claim 1 , wherein the data storage cluster is further configured to convey a write completion indication that corresponds to a received write request to the client, wherein said write completion indication is conveyed prior to completing performance of the write request in the array.
7. The system as recited in claim 6 , wherein the write request corresponds to a chosen particular row and one or more particular columns of the array, and wherein the data storage cluster is further configured to:
read old data only from storage locations targeted by the write request;
compute a new parity value for the chosen row based upon the old data, data that corresponds to the write request, and a stored old parity value corresponding to the old data.
8. A method comprising:
receiving read and write requests from a client in a data storage cluster, wherein the data storage cluster comprises a plurality of data storage locations addressable as rows and columns in an array;
in response to detecting a failure of a first column:
designating a second column as a replacement for the first column;
initiating reconstruction of first data stored in the first column; and
storing reconstructed data corresponding to the first data in the second column;
servicing received read and write requests during said reconstruction, whereby read data is returned to clients responsive to received read requests and/or write data is written to the array responsive to received write requests; and
storing information corresponding to a write request targeted to the second column in a temporary log during said reconstruction.
9. The method as recited in claim 8 , wherein each column of the array comprises a separate computer interconnected with other columns of the array via a network.
10. The method as recited in claim 8 , wherein during said reconstruction, the method further comprises:
receiving a read request targeted to a given row; and
servicing columns targeted by the read request, except for the second column.
11. The method as recited in claim 10 , further comprising designating the second column as a permanent replacement for the first column.
12. The method as recited in claim 11 , further comprising designating the second column as the permanent replacement in response to detecting a predetermined amount of time has elapsed after detecting the failure of the first column.
13. The method as recited in claim 8 , further comprising conveying a write completion indication that corresponds to a received write request to the client, wherein said write completion indication is conveyed prior to completing performance of the write request in the array.
14. The method as recited in claim 13 , wherein the array comprises a redundant array of independent disks (RAID) 5 layout.
15. A computer-readable storage medium storing program instructions that are executable to:
receive read and write requests from a client in a data storage cluster, wherein the data storage cluster comprises a plurality of data storage locations addressable as rows and columns in an array;
in response to detecting a failure of a first column:
designate a second column as a replacement for the first column;
initiate reconstruction of first data stored in the first column; and
store reconstructed data corresponding to the first data in the second column;
service received read and write requests during said reconstruction, whereby read data is returned to clients responsive to received read requests and/or write data is written to the array responsive to received write requests; and
store information corresponding to a write request targeted to the second column in a temporary log during said reconstruction.
16. The computer-readable storage medium as recited in claim 15 , wherein during said reconstruction, the program instructions are further executable to:
receive a read request targeted to a given row; and
service columns targeted by the read request, except for the second column.
17. The computer-readable storage medium as recited in claim 15 , wherein the program instructions are further executable to convey a write completion indication that corresponds to a received write request to the client, wherein said write completion indication is conveyed prior to completing performance of the write request in the array.
18. The computer-readable storage medium as recited in claim 15 , wherein the program instructions are further executable to designating the second column as a permanent replacement for the first column in response to detecting a predetermined amount of time has elapsed after detecting the failure of the first column.