Reliability coding with reduced network traffic
This disclosure describes techniques that include implementing network-efficient data durability or data reliability coding on a network. In one example, this disclosure describes a method that includes generating a plurality of data fragments from a set of data to enable reconstruction of the set of data from a subset of the plurality of data fragments; storing, across a plurality of nodes in a network, the plurality of data fragments, wherein storing the plurality of data fragments includes storing the first fragment at a first node and the second fragment at a second node; and generating, by the first node, a plurality of secondary fragments derived from the first fragment to enable reconstruction of the first fragment from a subset of the plurality of secondary fragments; and storing the plurality of secondary fragments from the first fragment across a plurality of storage devices included within the first node.
1. A method comprising:
generating a plurality of data fragments from a set of data to enable reconstruction of a portion of the data from a subset of the plurality of data fragments pursuant to a first level of reliability coding, wherein the plurality of data fragments includes a first fragment and a second fragment;
storing, across a plurality of nodes in a network, the plurality of data fragments, wherein storing the plurality of data fragments includes storing the first fragment at a first node and the second fragment at a second node;
generating, by the first node, a plurality of secondary fragments derived from the first fragment to enable reconstruction of the first fragment from a subset of the plurality of secondary fragments pursuant to a second level of reliability coding;
storing the plurality of secondary fragments from the first fragment across a plurality of storage devices included within the first node, wherein storing the plurality of secondary fragments includes storing each of the plurality of secondary fragments in a different one of the plurality of storage devices; and
reconstructing the portion of the data from a subset of the plurality of data fragments, wherein reconstructing the portion of the data includes coordinating the first level of reliability coding and the second level of reliability coding to reduce network traffic by reconstructing the first fragment from a subset of the plurality of secondary fragments and avoiding use of the plurality of data fragments stored across the plurality of nodes in the network.
2. The method of claim 1 , wherein reconstructing the portion of the data includes:
identifying a failed storage device within the first node, wherein the failed storage device is one of the plurality of storage devices included within the first node; and
reconstructing data stored at the failed storage device using secondary fragments stored across a reconstruction subset of the plurality of storage devices, wherein the reconstruction subset does not include the failed storage device.
3. The method of claim 2 , wherein reconstructing the data stored at the failed storage device includes:
reconstructing the data stored at the failed storage device within the first node without requiring network access to any of the other plurality of nodes in the network.
4. The method of claim 2 , wherein reconstructing the data stored at the failed storage device includes:
reconstructing the data stored at the failed storage device within the first node without generating network activity.
5. The method of claim 1 , wherein generating a plurality of data fragments from a set of data includes:
accumulating data from one or more servers; and
responsive to the set of data reaching a sufficient size, generating the plurality of data fragments from the set of data.
6. The method of claim 1 , wherein the plurality of storage devices is a first plurality of storage devices, and wherein the method further comprises:
generating, by the second node, a plurality of secondary fragments derived from the second fragment to enable reconstruction of the second fragment from a subset of the plurality of secondary fragments derived from the second fragment; and
storing the plurality of secondary fragments derived from the second fragment across a second plurality of storage devices included within the second node, wherein storing the plurality of secondary fragments derived from the second fragment includes storing each of the plurality of secondary fragments derived from the second fragment in a different one of the second plurality of storage devices.
7. The method of claim 6 , wherein reconstructing the portion of the data includes:
reconstructing the second fragment from a subset of the plurality of second fragments derived from the second fragment.
8. The method of claim 1 , wherein generating the plurality of data fragments includes:
generating a plurality of Reed Solomon erasure coding data fragments, the plurality of Reed Solomon erasure coding data fragments including a plurality of parity data fragments.
9. The method of claim 1 , wherein generating the plurality of secondary fragments from the first fragment includes:
generating one or more parity data fragments.
10. The method of claim 1 , wherein storing the plurality of data fragments includes:
storing each of the plurality of data fragments in a different fault domain.
11. The method of claim 10 , wherein the fault domain is defined as a node.
12. The method of claim 10 , wherein the fault domain is defined as a rack.
13. A storage system comprising:
a memory system; and
processing circuitry having access to the memory system, wherein the processing circuitry is configured to be capable of performing operations comprising:
generating a plurality of data fragments from a set of data to enable reconstruction of the set of data from a subset of the plurality of data fragments pursuant to a first level of reliability coding, wherein the plurality of data fragments includes a first fragment and a second fragment;
storing, across the plurality of nodes in a network, the plurality of data fragments, wherein storing the plurality of data fragments includes storing the first fragment at a first node and the second fragment at a second node, wherein the first node and the second node are included within the plurality of nodes;
generating a plurality of secondary fragments derived from the first fragment to enable reconstruction of the first fragment from a subset of the plurality of secondary fragments pursuant to a second level of reliability coding;
storing the plurality of secondary fragments from the first fragment across a plurality of storage devices included within the first node, wherein storing the plurality of secondary fragments includes storing each of the plurality of secondary fragments in a different one of the plurality of storage devices; and
reconstructing the set of data from a subset of the plurality of data fragments, wherein reconstructing the set of data includes coordinating the first level of reliability coding and the second level of reliability coding to reduce network traffic by reconstructing the first fragment from a subset of the plurality of secondary fragments and avoiding use of the plurality of data fragments stored across the plurality of nodes in the network.
14. The storage system of claim 13 , wherein to reconstruct the set of data, the storage system is further configured to perform operations comprising:
identifying a failed storage device within the first node, wherein the failed storage device is one of the plurality of storage devices included within the first node; and
reconstructing data stored at the failed storage device using secondary fragments stored across a reconstruction subset of the plurality of storage devices, wherein the reconstruction subset does not include the failed storage device.
15. The storage system of claim 14 , wherein to reconstruct the data stored at the failed storage device, the storage system is further configured to perform operations comprising:
reconstructing the data stored at the failed storage device within the first node without generating network activity.
16. The storage system of claim 14 , wherein to reconstruct the data stored at the failed storage device, the storage system is further configured to perform operations comprising:
reconstructing the data stored at the failed storage device within the first node without requiring network access to any of the other plurality of nodes in the network.
17. The storage system of claim 13 , wherein to generate a plurality of data fragments from a set of data, the storage system is further configured to perform operations comprising:
accumulating data from one or more devices; and
responsive to the set of data reaching a sufficient size, generating the plurality of data fragments from the set of data.
18. The storage system of claim 13 , wherein the plurality of storage devices is a first plurality of storage devices, and wherein the storage system is further configured to perform operations comprising:
generating, by the second node, a plurality of secondary fragments derived from the second fragment to enable reconstruction of the second fragment from a subset of the plurality of secondary fragments derived from the second fragment; and
storing the plurality of secondary fragments derived from the second fragment across a second plurality of storage devices included within the second node, wherein storing the plurality of secondary fragments derived from the second fragment includes storing each of the plurality of secondary fragments derived from the second fragment in a different one of the second plurality of storage devices.
19. The storage system of claim 18 , wherein to reconstruct the set of data, the storage system is further configured to perform operations comprising:
reconstructing the second fragment from a subset of the plurality of second fragments derived from the second fragment.
20. A non-transitory computer-readable storage medium comprising instructions that, when executed, configure processing circuitry of a storage system to perform operations comprising:
generating a plurality of data fragments from a set of data to enable reconstruction of at least a portion of the data from a subset of the plurality of data fragments pursuant to a first level of reliability coding, wherein the plurality of data fragments includes a first fragment and a second fragment;
storing, across a plurality of nodes in a network, the plurality of data fragments, wherein storing the plurality of data fragments includes storing the first fragment at a first node and the second fragment at a second node;
generating, by the first node, a plurality of secondary fragments derived from the first fragment to enable reconstruction of the first fragment from a subset of the plurality of secondary fragments pursuant to a second level of reliability coding;
storing the plurality of secondary fragments from the first fragment across a plurality of storage devices included within the first node, wherein storing the plurality of secondary fragments includes storing each of the plurality of secondary fragments in a different one of the plurality of storage devices; and
reconstructing the at least a portion of the data from a subset of the plurality of data fragments, wherein reconstructing the at least a portion of the data includes coordinating the first level of reliability coding and the second level of reliability coding to reduce network traffic by reconstructing the first fragment from a subset of the plurality of secondary fragments and avoiding use of the plurality of data fragments stored across the plurality of nodes in the network.