Service-level rebuild
Techniques are provided for service-level rebuild of one or more storage drives. Storage drives can store data of varying priority levels. In an example, when a storage drive is rebuilt, higher-priority data is written to the rebuilt storage drive before lower-priority data. In some examples where multiple storage drives are being rebuilt, a system can prioritize a use of rebuild bandwidth in rebuilding storage drives.
1. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the first processor, facilitate performance of operations, comprising:
determining to rebuild a failed storage drive with a first storage drive of a group of storage drives;
determining first data and second data to write to the first storage drive, the first data having a first priority, the second data having a second priority, and the first priority being greater than the second priority; and
based on the first priority being greater than the second priority and based on a prioritized use of a build bandwidth of the system corresponding to a determined risk that the set of storage drives has experienced data loss, writing the first data to the first storage drive before writing the second data to the first storage drive.
2. The system of claim 1 , wherein the group of storage drives is a first consistency group, and wherein the operations further comprise:
prioritizing use of a rebuild bandwidth of the system between the first consistency group and a second consistency group based on a first determined risk of the first consistency group having first data loss, and a second determined risk of the second consistency group having second data loss.
3. The system of claim 2 , wherein the operations further comprise:
modifying the prioritizing the use of the rebuild bandwidth of the system based on a first change to the first determined risk or a second change to the second determined risk.
4. The system of claim 2 , wherein the first determined risk is determined based on a degraded bytes of the first consistency group, and a degraded input/output rate of the first consistency group.
5. The system of claim 4 , wherein the first determined risk is determined based on an exponential moving average of the degraded bytes and of the degraded input/output rate.
6. The system of claim 2 , wherein the first determined risk is determined based on a combination of short-term degraded bytes of the first consistency group, and long-term degraded bytes of the first consistency group.
7. The system of claim 2 , wherein the operations further comprise:
resetting the first determined risk based on the first consistency group transitioning from a protected state to a degraded state.
8. The system of claim 2 , wherein the first determined risk is initially based on a third determined risk of a third consistency group.
9. A method, comprising:
determining, by a system comprising a processor, to rebuild a failed storage drive with a first storage drive of storage drives;
determining, by the system, first data and second data to write to the first storage drive; and
writing, by the system, the first data to the first storage drive before writing the second data to the first storage drive, based on a first priority of the first data being determined to be greater than a second priority of the second data, and based on a prioritized use of a build bandwidth of the system, wherein the prioritized use is determined as a function of a determined risk that the set of storage drives has a data loss.
10. The method of claim 9 , further comprising:
in response to determining that the writing the first data to the first storage drive has not been performed within the first defined amount of time, increasing, by the system, a prioritization of rebuild bandwidth of the system to the writing the first data to the first storage drive.
11. The method of claim 9 , further comprising:
predicting, by the system, a likelihood of the rebuilding the first storage drive based on a Markov model.
12. The method of claim 11 , wherein the Markov model comprises states that indicate that the storage drives are fully protected, partially protected, unprotected, degraded, and critically degraded.
13. The method of claim 11 , further comprising:
populating, by the system, the Markov model with samples of degraded tracks of the first storage drive.
14. The method of claim 9 , further comprising:
predicting, by the system, a likelihood of the rebuilding the first storage drive based on monitoring physical inconsistent tracks of the storage drives over time.
15. A non-transitory computer-readable medium comprising instructions that, in response to execution, cause a system comprising a processor to perform operations, comprising:
determining to rebuild a failed storage drive with a first storage drive of a set of storage drives;
determining first data and second data to write to the first storage drive, the first data having a first priority, and the second data having a second priority; and
based on the first priority being determined to be greater than the second priority, writing the first data to the first storage drive before writing the second data to the first storage drive based on a prioritized use of a build bandwidth of the system based on a determined risk of the set of storage drives having data loss.
16. The non-transitory computer-readable medium of claim 15 , wherein the first priority is independent of a location on the first storage drive where the first data is written.
17. The non-transitory computer-readable medium of claim 15 , wherein the writing the first data to the first storage drive is prioritized so that it is performed within a first defined amount of time.
18. The non-transitory computer-readable medium of claim 15 , wherein the determining to rebuild the first storage drive comprises:
determining to rebuild data to the first storage drive using data stored on the set of storage drives, or by copying data from at least a second storage drive of the set of storage drives.
19. The non-transitory computer-readable medium of claim 15 , wherein the set of storage drives comprises a redundant array of independent disks (RAID) array.
20. The non-transitory computer-readable medium of claim 15 , wherein the set of storage drives comprises a second storage drive that contains first information that backs up second information of the first storage drive or a third storage drive of the set of storage drives, and wherein the set of storage drives are able to rebuild the second information based on the first information when the second information is lost.