Data object recovery for storage systems
A method comprising: receiving an I/O request for object data; determining one or more data fragments wherein the object data is stored; determining that one or more of the data fragments are unavailable; determining, from within the one or more unavailable data fragments, a set of slices storing the object data, each slice comprising k small data fragments and m coded fragments; for each slice, retrieving at least k small data and coded fragments within the slice from storage; and recovering a segment of the object data using the retrieved small data and coded fragments. A related system and computer program product are also described.
1. A method for retrieving data in a storage system, the method comprising:
receiving, at the storage system, an I/O request for object data, the I/O request being received from a client device;
identifying one or more data fragments where the object data is stored;
determining, by the storage system, that a first one of the data fragments is unavailable due to a failure of a node in the storage system, the first data fragment containing a plurality of first small data fragments that constitute a first portion of the object data;
retrieving, a plurality of second small data fragments and a plurality of coded fragments, the plurality of second small data fragments being generated based on the plurality of second small data fragments and a coding matrix, the plurality of second small data fragments and the plurality of coded fragments being part of a slice that is associated with the first data fragment;
recovering, by the storage system, the first portion of the object data by using the plurality of second small data fragments and the plurality of coded fragments; and
completing, by the storage system, the I/O request for object data based on the recovered first portion of the object data.
2. The method of claim 1 , further comprising:
determining, that a second one of the data fragments is available;
retrieving the second data fragment; and
extracting a second portion of the object data from the second data fragment,
wherein the I/O request is completed further based on the extracted second portion of the object data.
3. The method of claim 1 , wherein recovering the first portion of the object data comprises performing erasure decoding on the slice.
4. The method of claim 1 , wherein the plurality of second small data fragments is retrieved from two or more different nodes within the storage system.
5. The method of claim 1 , wherein each of the data fragments is greater than 100 megabytes and each of the first small data fragments is less than 1 kilobyte.
6. The method of claim 5 , wherein each of the first small data fragments is a half-byte.
7. A storage system comprising:
one or more processors;
a volatile memory; and
a non-volatile memory storing computer program code that when executed on the processor causes execution across the one or more processors of a process for retrieving data comprising the operations of:
receiving an I/O request for object data, the I/O request being received from a client device;
identifying one or more data fragments where the object data is stored;
determining that a first one of the data fragments is unavailable due to a failure of a node in the storage system, the first data fragment containing a plurality of first small data fragments that constitute a first portion of the object data;
retrieving a plurality of second small data fragments and a plurality of coded fragments, the plurality of second small data fragments being generated based on the plurality of second small data fragments and a coding matrix, the plurality of second small data fragments and the plurality of coded fragments being part of a slice that is associated with the first data fragment;
recovering the first portion of the object data by using the plurality of second small data fragments and the plurality of coded fragments; and
completing the I/O request for object data based on the recovered first portion of the object data.
8. The storage system of claim 7 , wherein the computer program code causes execution across the one or more processors of the process further operable to perform the operations of:
determining that a second one of the data fragments is available;
retrieving the second data fragment; and
extracting a second portion of the object data from the retrieved second data fragment,
wherein the I/O request is completed further based on the extracted second portion of the object data.
9. The storage system of claim 7 , wherein recovering the first portion of the object data comprises performing erasure decoding on the slice.
10. The storage system of claim 7 , wherein each of the data fragments is greater than 100 megabytes and each of the first small data fragments is less than 1 kilobyte.
11. The storage system of claim 10 , wherein each of the first small data fragments is a half-byte.
12. A computer program product tangibly embodied in a non-transitory computer-readable medium, the computer-readable medium storing program instructions which when executed by at least one processor that is part of a storage system cause the at least one processor to:
receive an I/O request for object data that is transmitted to the storage system by a client device;
identify one or more data fragments where the object data is stored;
determine that a first one of the data fragments is unavailable due to a failure of a node in the storage system, the first data fragment containing a plurality of first small data fragments that constitute a first portion of the object data;
retrieving, a plurality of second small data fragments and a plurality of coded fragments, the plurality of second small data fragments being generated based on the plurality of second small data fragments and a coding matrix, the plurality of second small data fragments and the plurality of coded fragments being part of a slice that is associated with the first data fragment;
recover the first portion of the object data by using the plurality of second small data fragments and the plurality of coded fragments; and
complete the I/O request for object data based on the recovered first portion of the object data.
13. The computer program product of claim 12 , wherein the program instructions are further executable to:
determine that a second one of the data fragments is available;
retrieve the second data fragment; and
extract a second portion of the object data from the retrieved second data fragment,
wherein the I/O request is completed further based on the extracted second portion of the object data.
14. The computer program product of claim 12 wherein recovering the first portion of the object data comprises performing erasure decoding on the retrieved slices.
15. The computer program product of claim 12 , wherein the plurality of second small data fragments is retrieved from two or more different nodes within the storage system.
16. The computer program product of claim 12 , wherein each of the data fragments is greater than 100 megabytes and each of the first small data fragments is less than 1 kilobyte.
17. The computer program product of claim 16 , wherein each of the first small data fragments is a half-byte.