Redistributing data in a distributed storage system based on attributes of the data
Accesses to a number of data blocks stored in a distributed storage are observed. Following observation of the accesses, the stored data blocks are redistributed. In one aspect, redistribution of the data blocks includes determining the access patterns for one or more of the data blocks based on the observed accesses, and determining the storage sizes for the one or more data blocks. Thereafter, based on the determined access patterns and determined storage sizes, the one or more data blocks are sorted. Subsequently, the one or more data blocks are redistributed or rebalanced across a number of storage devices of the distributed storage based on the sorting. In one aspect, the one or more data blocks are redistributed according to either a uniform distribution scheme or a proportional distribution scheme.
1. A method comprising:
categorizing a plurality of data blocks into a plurality of categories according to an access pattern and a size corresponding to respective data blocks of the plurality of data blocks, wherein each category of the plurality of categories is associated with a respective access pattern level and respective data block storage size requirement;
calculating a target number of data blocks for a first category for each of a plurality of storage devices of a distributed storage by dividing a total number of data blocks in the first category by a number of the plurality of storage devices; and
redistributing the data blocks in the first category across the plurality of storage devices according to the calculated target number of data blocks for each of the plurality of storage devices.
2. The method of claim 1 , wherein redistributing comprises uniformly redistributing data blocks having similar access patterns and storage sizes across the plurality of storage devices.
3. The method of claim 1 , wherein redistributing is further based on a determined performance characteristic for the plurality of storage devices.
4. The method of claim 3 , wherein redistributing comprises redistributing the plurality of data blocks across the plurality of storage devices in proportion to a determined performance characteristic for the plurality of storage devices.
5. The method of claim 4 , wherein the determined performance characteristic comprises bandwidth of each of the plurality of storage devices.
6. The method of claim 4 , wherein the determined performance characteristic comprises storage capacity of each of the plurality of storage devices.
7. A non-transitory computer readable storage medium executing computer program instructions, the computer program instructions comprising instructions for:
categorizing a plurality of data blocks into a plurality of categories according to an access pattern and a size corresponding to respective data blocks of the plurality of data blocks, wherein each category of the plurality of categories is associated with a respective access pattern level and respective data block storage size requirement;
calculating a target number of data blocks for a first category for each of a plurality of storage devices of a distributed storage by dividing a total number of data blocks in the first category by a number of the plurality of storage devices; and
redistributing the data blocks in the first category across the plurality of storage devices according to the calculated target number of data blocks for each of the plurality of storage devices.
8. The medium of claim 7 , wherein redistributing comprises uniformly redistributing data blocks having similar access patterns and storage sizes across the plurality of storage devices.
9. The medium of claim 7 , wherein redistributing is further based on a determined performance characteristic for the plurality of storage devices.
10. The medium of claim 9 , wherein redistributing comprises redistributing the plurality of data blocks across the plurality of storage devices in proportion to the determined performance characteristic for the plurality of storage devices.
11. The medium of claim 10 , wherein the determined performance characteristic comprises bandwidth of each of the plurality of storage devices.
12. The medium of claim 10 , wherein the determined performance characteristic comprises storage capacity of each of the plurality of storage devices.
13. A system comprising:
a non-transitory computer readable storage medium storing processor-executable computer program instructions, the instructions comprising instructions for:
categorizing a plurality of data blocks into a plurality of categories according to an access pattern and a size corresponding to respective data blocks of the plurality of data blocks, wherein each category of the plurality of categories is associated with a respective access pattern level and respective data block storage size requirement;
calculating a target number of data blocks for a first category for each of a plurality of storage devices of a distributed storage by dividing a total number of data blocks in the first category by a number of the plurality of storage devices; and
redistributing the data blocks in the first category across the plurality of storage devices according to the calculated target number of data blocks for each of the plurality of storage devices; and
a processor for executing the computer program instructions.
14. The system of claim 13 , wherein redistributing comprises uniformly redistributing data blocks having similar access patterns and storage sizes across the plurality of storage devices.
15. The system of claim 13 , wherein redistributing is further based on a determined performance characteristic for the plurality of storage devices.
16. The system of claim 15 , wherein redistributing comprises redistributing the plurality of data blocks across the plurality of storage devices in proportion to the determined performance characteristic for the plurality of storage devices.
17. The system of claim 16 , wherein the determined performance characteristic comprises bandwidth of each of the plurality of storage devices.
18. The system of claim 16 , wherein the determined performance characteristic comprises storage capacity of each of the plurality of storage devices.