Adaptive rebuilding rates based on sampling and inference
A method for execution by one or more processing modules of a dispersed storage network (DSN), the method begins by monitoring an encoded data slice access rate to produce an encoded data slice access rate for an associated rebuilding rate of a set of rebuilding rates. The method continues by applying a learning function to the encoded data slice access rate based on a previous encoded data slice access rate associated with the rebuilding rate to produce an updated previous encoded data slice access rate of a set of previous encoded data slice access rates. The method continues by updating a score value associated with the updated previous encoded data slice access rate and the rebuilding rate and selecting a slice access scheme based on the updated score value where a rebuild rate selection will maximize a score value associated with an expected slice access rate.
1. An integrity processing unit for determining a rate to rebuild encoded data slices stored within a dispersed storage network (DSN), the integrity processing unit comprises:
an interface;
a local memory; and
a processing module operably coupled to the interface and the local memory, wherein the processing module functions to:
at each time interval T, select a rebuild rate at which to rebuild data from a range of possible rebuild rates;
calculate a score value based on the rebuild rate and an I/O rate, the calculating providing a score value for each of a set of possible rebuild rates;
find a sum total of all the score values for each score value in the set of possible rebuild rates;
divide each score value by this sum total to get weighted probabilities; and
for each time interval T, selecting at least one of the weighted probabilities for a selection of a next rebuild rate to use; and
rebuild the data in DSN storage using the next rebuild rate.
2. The integrity processing unit of claim 1 , wherein the select a rebuild rate at which to rebuild data from a range of possible rebuild rates includes monitoring an aggregate rate of I/O operations processed in T.
3. The integrity processing unit of claim 1 , wherein the calculate a score value includes calculating ((N*rebuild rate)+I/O rate) 2 , where N is a multiplier of the rebuild rate.
4. The integrity processing unit of claim 1 further comprises the score value biasing a selection of the next rebuild rate to use.
5. The integrity processing unit of claim 1 , wherein the score value biasing a selection of the next rebuild rate to use includes a bias towards the rebuild rate that results in a highest calculated score value.
6. The integrity processing unit of claim 5 , wherein a sum of all the weighted probabilities is 100%.
7. The integrity processing unit of claim 1 , wherein the score value is calculated as:
((3*rebuild rate)+encoded data slice access rate) 2 .
8. The integrity processing unit of claim 1 further comprises including a learning rate, where for a given selected rebuilding rate, an associated score value may be subsequently updated in accordance with a learning rate function when an updated corresponding slice access rate is measured for the given selected rebuilding rate.
9. The integrity processing unit of claim 8 , wherein the learning rate function is calculated as:
updated score=(old score)*(1−learning rate)+(new score*learning rate).
10. The integrity processing unit of claim 1 , wherein determining to update the rebuild rate for a storage unit is based on one or more of: detecting an end of a time interval, receiving a request, receiving an error message, or detecting an unfavorable slice access rate.
11. The integrity processing unit of claim 1 , wherein determining a slice access demand rate and rebuilding access demand rate is based on one or more of: interpreting a queue, receiving a request, or accessing a historical record.
12. The integrity processing unit of claim 1 , wherein selecting an encoded data slice access scheme is based on one or more of: a predetermination, detecting that a demand rate is much greater than a demand threshold level, or receiving a request.
13. A computing device of a group of computing devices of a dispersed storage network (DSN), the computing device comprises:
an interface;
a local memory; and
a processing module operably coupled to the interface and the local memory, wherein the processing module functions to:
at each time interval T, select a rebuild rate at which to rebuild data from a range of possible rebuild rates;
calculate a score value based on the rebuild rate and an I/O rate, the calculating providing a score value for each of a set of possible rebuild rates;
find a sum total of all the score values for each score value in the set of possible rebuild rates;
divide each score value by this sum total to get weighted probabilities;
wherein, for each time interval T, the integrity processing unit uses the weighted probabilities for a selection of a next rebuild rate to use; and
rebuild the data in DSN storage using the next rebuild rate.
14. A computing device of claim 13 , wherein the select a rebuild rate at which to rebuild data from a range of possible rebuild rates is performed while monitoring an aggregate rate of I/O operations processed in T.
15. A computing device of claim 13 , wherein the score value calculated as ((N*rebuild rate)+I/O rate) 2 , where N is a multiplier of the rebuild rate.
16. A computing device of claim 13 further comprises the score value biasing a selection of the next rebuild rate to use.
17. A computing device of claim 13 , wherein the score value biasing a selection of the next rebuild rate to use includes a bias towards the rebuild rate that results in a highest calculated score value.
18. A computing device of claim 13 , wherein a sum of all the weighted probabilities is 100%.
19. A method to select a rebuild rate for a dispersed storage network (DSN), the method comprises:
at each time interval T, selecting a rebuild rate at which to rebuild data from a range of possible rebuild rates while monitoring an aggregate rate of I/O operations processed in T;
calculating a score value based on the rebuild rate and an I/O rate, the calculating providing a score value for each of a set of possible rebuild rates, the score value calculated as ((N*rebuild rate)+I/O rate) 2 , where N is a multiplier of the rebuild rate;
finding a sum total of all the score values for each score value in the set of possible rebuild rates;
dividing each score value by this sum total to get weighted probabilities; and
wherein, for each time interval T, selecting a highest of the weighted probabilities for a selection of a next rebuild rate to use; and
rebuilding the data in DSN storage using the next rebuild rate.
20. The method of claim 19 , wherein the score value biases a selection of the next rebuild rate towards based on a rebuild rate that result in a highest calculated score value.