System and method for managing backup operations of storage devices in a backup storage system using cluster evaluations based on usage telemetry
A method for managing storage devices includes obtaining a storage device cluster request, and in response to the storage device cluster request: obtaining a set of storage device telemetry entries associated with a plurality of storage devices, performing a telemetry normalization on the storage device telemetry entries to obtain a set of normalized entries, performing a pairwise evaluation on the set of normalized entries to obtain a set of initial storage device clusters, wherein a storage device cluster in the set of initial storage device clusters comprises a portion of the plurality of storage devices, performing a cluster re-evaluation on the set of initial storage device cluster groups to obtain a set of updated storage device clusters, updating a backup policy based on the set of updated storage device cluster groups, and performing a backup operation on a storage device based on the backup policy.
1. A method for managing storage devices in a backup storage system, the method comprising:
obtaining, by a backup agent, a storage device cluster request; and
in response to the storage device cluster request:
obtaining a set of storage device telemetry entries associated with a plurality of storage devices;
performing a telemetry normalization on the storage device telemetry entries to obtain a set of normalized entries;
performing a pairwise evaluation on the set of normalized entries to obtain a set of initial storage device clusters, wherein a storage device cluster in the set of initial storage device clusters comprises a portion of the plurality of storage devices;
performing a cluster re-evaluation on the set of initial storage device cluster groups to obtain a set of updated storage device clusters;
updating a backup policy based on the set of updated storage device cluster groups; and
performing a backup operation on a storage device of the plurality of storage devices based on the backup policy.
2. The method of claim 1 , wherein a storage device telemetry entry in the set of storage device telemetry entries comprises a time series of a variable in the set of variables as a function of time.
3. The method of claim 2 ,
wherein each normalized entry in the set of normalized entries comprises a portion of a storage device telemetry entry of the storage device telemetry entry, and
wherein each normalized entry in the set of normalized entries is a time series associated with an equal length of time.
4. The method of claim 1 , wherein performing the pairwise evaluation comprises:
selecting two normalized entries in the set of normalized entries;
calculating a pairwise distance between the two normalized entries; and
assigning a cluster group to a storage device associated with each of the two normalized entries based on the pairwise distance.
5. The method of claim 1 , wherein performing the cluster re-evaluation comprises:
determining a first portion of the set of storage device cluster groups based on a cluster group criterion; and
assigning each storage device of a second portion of the set of cluster groups to a storage device cluster group of the first portion of the set of storage device cluster groups,
wherein each storage device cluster group of the second portion of the set of storage device cluster groups does not meet the cluster group criterion.
6. The method of claim 1 , wherein updating the backup policy comprises:
modifying a schedule of the backup policy based on at least one storage device cluster of the storage device cluster,
wherein the schedule is determined based on a period of time required to perform a backup operation.
7. The method of claim 1 , wherein the set of storage devices are grouped into storage device pools.
8. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for managing storage devices in a backup storage system, the method comprising:
obtaining, by a backup agent, a storage device cluster request; and
in response to the storage device cluster request:
obtaining a set of storage device telemetry entries associated with a plurality of storage devices;
performing a telemetry normalization on the storage device telemetry entries to obtain a set of normalized entries;
performing a pairwise evaluation on the set of normalized entries to obtain a set of initial storage device clusters, wherein a storage device cluster in the set of initial storage device clusters comprises a portion of the plurality of storage devices;
performing a cluster re-evaluation on the set of initial storage device cluster groups to obtain a set of updated storage device clusters;
updating a backup policy based on the set of updated storage device cluster groups; and
performing a backup operation on a storage device of the plurality of storage devices based on the backup policy.
9. The non-transitory computer readable medium of claim 8 , wherein a storage device telemetry entry in the set of storage device telemetry entries comprises a time series of a variable in the set of variables as a function of time.
10. The non-transitory computer readable medium of claim 8 ,
wherein each normalized entry in the set of normalized entries comprises a portion of a storage device telemetry entry of the storage device telemetry entry, and
wherein each normalized entry in the set of normalized entries is a time series associated with an equal length of time.
11. The non-transitory computer readable medium of claim 8 , wherein performing the pairwise evaluation comprises:
selecting two normalized entries in the set of normalized entries;
calculating a pairwise distance between the two normalized entries; and
assigning a cluster group to a storage device associated with each of the two normalized entries based on the pairwise distance.
12. The non-transitory computer readable medium of claim 8 , wherein performing the cluster re-evaluation comprises:
determining a first portion of the set of storage device cluster groups based on a cluster group criterion; and
assigning each storage device of a second portion of the set of cluster groups to a storage device cluster group of the first portion of the set of storage device cluster groups,
wherein each storage device cluster group of the second portion of the set of storage device cluster groups does not meet the cluster group criterion.
13. The non-transitory computer readable medium of claim 8 , wherein updating the backup policy comprises:
modifying a schedule of the backup policy based on at least one storage device cluster of the storage device cluster,
wherein the schedule is determined based on a period of time required to perform a backup operation.
14. The non-transitory computer readable medium of claim 8 , wherein the set of storage devices are grouped into storage device pools.
15. A system, comprising:
a processor; and
memory comprising instructions which, when executed by the processor, perform a method, the method comprising:
obtaining, by a backup agent, a storage device cluster request; and
in response to the storage device cluster request:
obtaining a set of storage device telemetry entries associated with a plurality of storage devices;
performing a telemetry normalization on the storage device telemetry entries to obtain a set of normalized entries;
performing a pairwise evaluation on the set of normalized entries to obtain a set of initial storage device clusters, wherein a storage device cluster in the set of initial storage device clusters comprises a portion of the plurality of storage devices;
performing a cluster re-evaluation on the set of initial storage device cluster groups to obtain a set of updated storage device clusters;
updating a backup policy based on the set of updated storage device cluster groups; and
performing a backup operation on a storage device of the plurality of storage devices based on the backup policy.
16. The system of claim 15 , wherein a storage device telemetry entry in the set of storage device telemetry entries comprises a time series of a variable in the set of variables as a function of time.
17. The system of claim 16 ,
wherein each normalized entry in the set of normalized entries comprises a portion of a storage device telemetry entry of the storage device telemetry entry, and
wherein each normalized entry in the set of normalized entries is a time series associated with an equal length of time.
18. The system of claim 15 , wherein performing the pairwise evaluation comprises:
selecting two normalized entries in the set of normalized entries;
calculating a pairwise distance between the two normalized entries; and
assigning a cluster group to a storage device associated with each of the two normalized entries based on the pairwise distance.
19. The system of claim 15 wherein performing the cluster re-evaluation comprises:
determining a first portion of the set of storage device cluster groups based on a cluster group criterion; and
assigning each storage device of a second portion of the set of cluster groups to a storage device cluster group of the first portion of the set of storage device cluster groups,
wherein each storage device cluster group of the second portion of the set of storage device cluster groups does not meet the cluster group criterion.
20. The system of claim 15 , wherein updating the backup policy comprises:
modifying a schedule of the backup policy based on at least one storage device cluster of the storage device cluster,
wherein the schedule is determined based on a period of time required to perform a backup operation.