Reliability coding with reduced network traffic
This disclosure describes techniques that include implementing network-efficient data durability or data reliability coding on a network. In one example, this disclosure describes a method that includes generating a plurality of data fragments from data to enable reconstruction of the data from a subset of the plurality of data fragments; storing, across a plurality of nodes in a network, the plurality of data fragments, wherein storing the plurality of data fragments includes storing the first fragment at a first node and the second fragment at a second node; and generating, by the first node, a plurality of secondary fragments derived from the first fragment to enable reconstruction of the first fragment from a subset of the plurality of secondary fragments; and storing the plurality of secondary fragments from the first fragment across a plurality of storage devices included within the first node.
1 . A system comprising processing circuitry configured to:
generate a plurality of data fragments from data to enable reconstruction of a portion of the data from a subset of the plurality of data fragments;
store a first fragment of the plurality of data fragments at a first node in a network;
generate a plurality of secondary fragments derived from the first fragment to enable reconstruction of the first fragment from a subset of the plurality of secondary fragments derived from the first fragment;
store the plurality of secondary fragments derived from the first fragment across a plurality of storage devices included within the first node; and
reconstruct the portion of the data by at least reconstructing the first fragment from a subset of the plurality of secondary fragments derived from the first fragment while avoiding use of data durability coding implemented network-wide in the network.
2 . The system of claim 1 , wherein to reconstruct the portion of the data, the processing circuitry is further configured to:
identify a failed storage device within the first node, wherein the failed storage device is one of the plurality of storage devices included within the first node; and
reconstruct data stored at the failed storage device using secondary fragments stored across a reconstruction subset of the plurality of storage devices, wherein the reconstruction subset does not include the failed storage device.
3 . The system of claim 2 , wherein to reconstruct the data stored at the failed storage device, the processing circuitry is further configured to:
reconstruct the data stored at the failed storage device within the first node without requiring network access to any of other nodes in the network.
4 . The system of claim 2 , wherein to reconstruct the data stored at the failed storage device, the processing circuitry is further configured to:
reconstruct the data stored at the failed storage device within the first node without generating network activity.
5 . The system of claim 1 , wherein to generate a plurality of data fragments from the data, the processing circuitry is further configured to:
accumulate data from one or more servers; and
responsive to the data reaching a sufficient size, generate the plurality of data fragments from the data.
6 . The system of claim 1 , wherein the plurality of storage devices is a first plurality of storage devices, and wherein the processing circuitry is further configured to:
generate a plurality of secondary fragments derived from a second fragment to enable reconstruction of the second fragment from a subset of the plurality of secondary fragments derived from the second fragment; and
store the plurality of secondary fragments derived from the second fragment across a second plurality of storage devices included within a second node in the network.
7 . The system of claim 6 , wherein to reconstruct the portion of the data, the processing circuitry is further configured to:
reconstruct the second fragment from a subset of the plurality of second fragments derived from the second fragment.
8 . The system of claim 1 , wherein to generate the plurality of data fragments, the processing circuitry is further configured to:
generate a plurality of Reed Solomon erasure coding data fragments, the plurality of Reed Solomon erasure coding data fragments including a plurality of parity data fragments.
9 . The system of claim 1 , wherein to generate the plurality of secondary fragments derived from the first fragment, the processing circuitry is further configured to:
generate one or more parity data fragments.
10 . The system of claim 1 , wherein the processing circuitry is further configured to:
store each of the plurality of data fragments in a different fault domain.
11 . The system of claim 10 , wherein the fault domain is defined as a node.
12 . The system of claim 10 , wherein the fault domain is defined as a rack.
13 . A method comprising:
generating a plurality of data fragments from data to enable reconstruction of the data from a subset of the plurality of data fragments;
storing a first fragment of the plurality of data fragments at a first node in a network;
generating a plurality of secondary fragments derived from the first fragment to enable reconstruction of the first fragment from a subset of the plurality of secondary fragments derived from the first fragment;
storing the plurality of secondary fragments derived from the first fragment across a plurality of storage devices included within the first node; and
reconstructing the portion of the data by at least reconstructing the first fragment from a subset of the plurality of secondary fragments derived from the first fragment while avoiding use of data durability coding implemented network-wide in the network.
14 . The method of claim 13 , wherein reconstructing the portion of the data includes:
identifying a failed storage device within the first node, wherein the failed storage device is one of the plurality of storage devices included within the first node; and
reconstructing data stored at the failed storage device using secondary fragments stored across a reconstruction subset of the plurality of storage devices, wherein the reconstruction subset does not include the failed storage device.
15 . The method of claim 14 , wherein reconstructing the data stored at the failed storage device includes:
reconstructing the data stored at the failed storage device within the first node without generating network activity.
16 . The method of claim 14 , wherein reconstructing the data stored at the failed storage device includes:
reconstructing the data stored at the failed storage device within the first node without requiring network access to any of other nodes in the network.
17 . The method of claim 13 , wherein generating a plurality of data fragments from data includes:
accumulating data from one or more devices; and
responsive to the data reaching a sufficient size, generating the plurality of data fragments from the data.
18 . The method of claim 13 , wherein the plurality of storage devices is a first plurality of storage devices, the method further comprising:
generating a plurality of secondary fragments derived from a second fragment to enable reconstruction of the second fragment from a subset of the plurality of secondary fragments derived from the second fragment; and
storing the plurality of secondary fragments derived from the second fragment across a second plurality of storage devices included within a second node in the network.
19 . The method of claim 18 , wherein reconstructing the data includes:
reconstructing the second fragment from a subset of the plurality of second fragments derived from the second fragment.
20 . A computer-readable storage media for storing computer-readable instructions that, when executed, configure processing circuitry of a storage system to perform operations comprising:
generating a plurality of data fragments from data to enable reconstruction of at least a portion of the data from a subset of the plurality of data fragments;
storing a first fragment of the plurality of data fragments at a first node in a network;
generating a plurality of secondary fragments derived from the first fragment to enable reconstruction of the first fragment from a subset of the plurality of secondary fragments derived from the first fragment;
storing the plurality of secondary fragments derived from the first fragment across a plurality of storage devices included within the first node; and
reconstructing the portion of the data by at least reconstructing the first fragment from a subset of the plurality of secondary fragments derived from the first fragment while avoiding use of data durability coding implemented network-wide in the network.