Data recovery and regeneration using parity code
The disclosed technology can advantageously provide an efficient data recovery system including a plurality of storage nodes including a first storage node and a second storage node, and a storage logic that is coupled to the storage nodes and that manages storage of data on the storage nodes. The storage logic is executable to: receive a data set including data elements including a first set of data elements associated with the first storage node and a second set of data elements associated with the second storage node; generate a first parity of the data set, the first parity including a horizontal parity including a set of horizontal parity entries; and combine the data elements from the data set to produce a skipper parity including a set of skipper parity entries. Combining the data elements includes transforming a subset of the data elements from the data set using an invertible operation, the set of horizontal parity entries being different from the set of skipper parity entries.
1. A data storage system comprising:
a plurality of storage nodes including a first storage node and a second storage node; and
memory storing storage logic that is coupled to the plurality of storage nodes and that manages storage of data on the plurality of storage nodes, the storage logic executable by a hardware processor to:
receive a data set including a plurality of data elements associated with the plurality of storage nodes, the data set including:
a first set of data elements associated with the first storage node; and
a second set of data elements associated with the second storage node;
generate a horizontal parity including a set of horizontal parity entries that correspond to row combinations of the plurality of data elements associated with the plurality of storage nodes;
store the horizontal parity on a third storage node, the third storage node being different from the plurality of storage nodes;
generate a skipper parity including a set of skipper parity entries by transforming a subset of the data elements from the data set using an invertible operation, the set of horizontal parity entries being different from the set of skipper parity entries;
store the skipper parity on the third storage node; and
responsive to determining a failure of the first storage node, recreating data elements of the first storage node using a portion of data elements associated with storage nodes of the plurality of storage nodes that have not failed in combination with the horizontal parity and the skipper parity.
2. The data storage system of claim 1 , wherein the invertible operation includes one of a shift operation and an exclusive-or (XOR) operation.
3. The data storage system of claim 1 , wherein the storage logic is further executable to:
receive a request to recreate the first set of data elements using the horizontal parity and the skipper parity.
4. The data storage system of claim 3 , wherein the first storage node is a failed storage node.
5. The data storage system of claim 3 , wherein the storage logic is further executable to:
in response to receiving the request to recreate the first set of data elements, retrieve the second set of data elements, the horizontal parity, and the skipper parity; and
recreate the first set of data elements based on the second set of data elements, the horizontal parity, and the skipper parity.
6. The data storage system of claim 3 , wherein the storage logic is further executable to:
in response to receiving the request to recreate the first set of data elements, determine needed data elements of the second storage node to recreate data for the first set of data elements, the needed data elements comprising a subset of the second set of data elements;
determine needed horizontal parity entries of the horizontal parity and needed skipper parity entries of the skipper parity; and
recreate the first set of data elements based on the needed data elements of the second storage node and the determined parity entries.
7. The data storage system of claim 6 , wherein the needed data elements of the second storage node include at least half and less than all of the second set of data elements.
8. The data storage system of claim 1 , wherein to combine the data elements from the data set to produce the skipper parity includes determining locations of certain skipper parity data elements in a skipper parity matrix using a predefined rule.
9. A computer-implemented method comprising:
receiving a data set comprising a plurality of data elements associated with a plurality of storage nodes, the data set including a first set of data elements associated with a first storage node of the plurality of storage nodes and a second set of data elements associated with a second storage node of the plurality of storage nodes;
generating a horizontal parity of the data set, the horizontal parity including a set of horizontal parity entries;
store the horizontal parity in a third storage node, the third storage node being different from the plurality of storage nodes;
generate a skipper parity including a set of skipper parity entries, by transforming a subset of the data elements from the data set using an invertible operation, the set of horizontal parity entries being different from the set of skipper parity entries;
store the skipper parity in the third storage node; and
responsive to determining a failure of the first storage node, recreate data elements of the first storage node using a portion of data elements associated with storage nodes of the plurality of storage nodes that have not failed in combination with the horizontal parity and the skipper parity.
10. The computer-implemented method of claim 9 , wherein the invertible operation includes one of a shift operation and an exclusive-or (XOR) operation.
11. The computer-implemented method of claim 9 , further comprising receiving a request to recreate the first set of data elements using the horizontal parity and the skipper parity.
12. The computer-implemented method of claim 11 , wherein the first storage node is a failed storage node.
13. The computer-implemented method of claim 11 , further comprising:
in response to receiving the request to recreate the first set of data elements, retrieving the second set of data elements, the horizontal parity, and the skipper parity; and
recreating the first set of data elements based on the second set of data elements, the horizontal parity, and the skipper parity.
14. The computer-implemented method of claim 11 , further comprising:
in response to receiving the request to recreate the first set of data elements, determining needed data elements of the second storage node to recreate data for the first set of data elements, the needed data elements comprising a subset of the second set of data elements;
determining needed horizontal parity entries of the horizontal parity and needed skipper parity entries of the skipper parity; and
recreating the first set of data elements based on the needed data elements of the second storage node and the determined parity entries.
15. The computer-implemented method of claim 14 , wherein the needed data elements of the second storage node include at least half and less than all of the second set of data elements.
16. The computer-implemented method of claim 9 , wherein combining the data elements from the data set to produce the skipper parity includes determining locations of certain skipper parity data elements in a skipper parity matrix using a predefined rule.
17. A system comprising:
means for receiving a data set comprising a plurality of data elements associated with a plurality of storage nodes, the data set including a first set of data elements associated with a first storage node of the plurality of storage nodes and a second set of data elements associated with a second storage node of the plurality of storage nodes;
means for generating a horizontal parity of the data set, the horizontal parity including a set of horizontal parity entries;
means for storing the horizontal parity in a third storage node, the third storage node being different from the plurality of storage nodes;
means for generating a skipper parity including a set of skipper parity entries generated by transforming a subset of the data elements from the data set using an invertible operation, the set of horizontal parity entries being different from the set of skipper parity entries.
18. The system of claim 17 , wherein the invertible operation includes one of a shift operation and an exclusive-or (XOR) operation.
19. The system of claim 17 , wherein the first storage node is a failed storage node.
20. The system of claim 17 , wherein combining the data elements from the data set to produce the skipper parity includes determining locations of certain skipper parity data elements in a skipper parity matrix using a predefined rule.