Assigning prioritized rebuild resources optimally
An integrity processing unit includes rebuild modules to rebuild one or more encoded data slices in a dispersed storage network (DSN) memory unit. The rebuild modules determine a rebuild rate of the DSN memory unit, and, based on the rebuild rate, a rebuild rate status of the DSN memory unit. When the rebuild rate status is a high rebuild rate status, the rebuild rate to the DSN memory unit is reduced and, when the rebuild rate is not zero, the one or more encoded data slices are rebuilt in the DSN memory unit.
1. A distributed storage network (DSN) comprises:
an integrity processing unit;
a set of DSN memory units coupled with the integrity processing unit;
a plurality of rebuild modules operative with the integrity processing unit to rebuild one or more encoded data slices stored in a DSN memory unit of the set of DNS memory units;
determining a rebuild rate of the DSN memory unit, wherein determining the rebuild rate includes determining:
a number of the plurality of rebuild modules concurrently rebuilding encoded data slices for the DSN memory unit, and
respective rebuild rates of the plurality of rebuild modules concurrently rebuilding encoded data slices for the DSN memory unit;
determining, by the integrity processing unit and based on the rebuild rate of the DSN memory unit, a rebuild rate status of the DSN memory unit,
wherein a high rebuild rate status results from a collected traffic rate/network capacity rate of the DSN memory unit being above a first threshold,
a low rebuild rate status results from the collected traffic rate/network capacity rate of the DSN memory unit being below a second threshold,
the collected traffic rate comprises at least a sum of the rebuild rate of the DSN memory unit and a regular access traffic rate of the DSN memory unit, and
the first threshold is greater than the second threshold; and
when the rebuild rate status is the high rebuild rate status, reducing the rebuild rate to the DSN memory unit such that the collected traffic rate/network capacity rate is below the first threshold, the reducing including one or more of: reducing the number of the plurality of rebuild modules concurrently rebuilding encoded data slices to the DSN memory unit and reducing the rebuild rates of one or more of the plurality of rebuild modules;
rebuilding, when the rebuild rate is not reduced to zero, the one or more encoded data slices to produce one or more rebuilt encoded data slices; and
sending, by the integrity processing unit, the one or more rebuilt encoded data slices to the DSN memory unit for storage.
2. The DSN of claim 1 , further comprising when the rebuild rate status is the low rebuild rate status, increasing the rebuild rate to the DSN memory unit such that the collected traffic rate/network capacity rate is above the second threshold and below the first threshold.
3. The DSN of claim 1 , further comprising when the rebuild rate status is the high rebuild rate status, reducing the rebuild rate to zero and delaying the rebuilding of the one or more encoded data slices.
4. The DSN of claim 1 , wherein the rebuilding the one or more encoded data slices is based on a prioritized rebuild queue.
5. The DSN of claim 4 , wherein the prioritized rebuild queue is based on one or more risk factors that determine an order in which the DSN memory units of the set of DSN memory units are to have their encoded data slices rebuilt.
6. The DSN of claim 5 , wherein the one or more risk factors are at least one selected from the group consisting of:
hardware class of a site of each of the DSN memory units,
hardware class of a dispersed storage (DS) unit of the DSN memory units,
hardware class of a memory device where the encoded data slice(s) will be rebuilt within of each of the DSN memory units,
age of a site of the DSN memory units,
age of the DS unit,
age of a memory device where the one or more encoded data slices will be rebuilt within the DSN memory units,
whether a site DS unit of each of the DSN memory units is subject to frequent power outages,
whether the DSN memory units are subject to frequent network connectivity outages, adverse weather conditions, natural disaster predictions, and
instability.
7. The DSN of claim 1 , wherein the rebuild rate is determined through passive means.
8. The DSN of claim 1 , wherein the rebuild rate is determined by requesting data directly from the DSN memory unit.
9. The DSN of claim 1 , wherein the DSN memory unit is a dispersed storage (DS) unit.
10. The DSN of claim 1 , wherein the DSN memory unit is a memory device.
11. An integrity processing unit comprises:
a plurality of rebuild modules to rebuild one or more encoded data slices stored in a DSN memory unit in a set of DNS memory units, wherein at least one of the plurality of rebuild modules:
determines a rebuild rate of the DSN memory unit, wherein determining the rebuild rate includes determining
a number of the rebuild modules concurrently rebuilding encoded data slices for the DSN memory unit and
respective rebuild rates of the plurality of rebuild modules concurrently rebuilding encoded data slices for the DSN memory unit;
determines, based on the rebuild rate of the DSN memory unit, a rebuild rate status of the DSN memory unit,
wherein a high rebuild rate status results from a collected traffic rate/network capacity rate of the DSN memory unit being above a first threshold,
a low rebuild rate status results from the collected traffic rate/network capacity rate of the DSN memory unit being below a second threshold,
the collected traffic rate comprises at least a sum of the rebuild rate of the DSN memory unit and a regular access traffic rate of the DSN memory unit, and
the first threshold is greater than the second threshold;
when the rebuild rate status is the high rebuild rate status, reducing the rebuild rate of the DSN memory unit such that the collected traffic rate/network capacity rate is below the first threshold, the reducing including one or more of: reducing the number of the rebuild modules concurrently rebuilding encoded data slices to the DSN memory unit and reducing the rebuild rates of one or more of the rebuild modules;
rebuilds, when the rebuild rate is not reduced to zero, the one or more encoded data slices to produce one or more rebuilt encoded data slices; and
sends the one or more rebuilt encoded data slices to the DSN memory unit for storage.
12. The integrity processing unit of claim 11 , further comprising when the rebuild rate status is the low rebuild rate status, increasing the rebuild rate to the DSN memory unit such that the collected traffic rate/network capacity rate is above the second threshold and below the first threshold.
13. The integrity processing unit of claim 11 , further comprising when the rebuild rate status is the high rebuild rate status, reducing the rebuild rate to zero and delaying the rebuilding of the one or more encoded data slices.
14. The integrity processing unit of claim 11 , wherein the rebuilding the one or more encoded data slices is based on a prioritized rebuild queue.
15. The integrity processing unit of claim 14 , wherein the prioritized rebuild queue is based on one or more risk factors that determine an order in which the DSN memory units of the set of DSN memory units are to have their encoded data slices rebuilt.
16. The integrity processing unit of claim 15 , wherein the one or more risk factors are at least one selected from the group consisting of:
hardware class of a site of each of the DSN memory units,
hardware class of a dispersed storage (DS) unit of the DSN memory units,
hardware class of a memory device where the encoded data slice(s) will be rebuilt within of each of the DSN memory units,
age of a site of the DSN memory units,
age of the DS unit,
age of a memory device where the one or more encoded data slices will be rebuilt within the DSN memory units,
whether a site DS unit of each of the DSN memory units is subject to frequent power outages,
whether the DSN memory units are subject to frequent network connectivity outages, adverse weather conditions, natural disaster predictions, and
instability.
17. The integrity processing unit of claim 11 , wherein the rebuild rate is determined through passive means.
18. The integrity processing unit of claim 11 , wherein the rebuild rate is determined by requesting data directly from the DSN memory unit.
19. The integrity processing unit of claim 11 , wherein the DSN memory unit is a dispersed storage (DS) unit.
20. The integrity processing unit of claim 11 , wherein the DSN memory unit is a memory device.