Redistributing data in a distributed storage system based on attributes of the data
Accesses to a number of data blocks stored in a distributed storage are observed. Following observation of the accesses, the stored data blocks are redistributed. In one aspect, redistribution of the data blocks includes determining the access patterns for one or more of the data blocks based on the observed accesses, and determining the storage sizes for the one or more data blocks. Thereafter, based on the determined access patterns and determined storage sizes, the one or more data blocks are sorted. Subsequently, the one or more data blocks are redistributed or rebalanced across a number of storage devices of the distributed storage based on the sorting. In one aspect, the one or more data blocks are redistributed according to either a uniform distribution scheme or a proportional distribution scheme.
1. A computer-implemented method for redistributing a plurality of data blocks stored in a distributed storage, the method comprising:
observing accesses to the plurality of data blocks stored in the distributed storage; and
redistributing the plurality of data blocks in the distributed storage, wherein redistribution of the plurality of data blocks in the distributed storage comprises:
determining access patterns for one or more data blocks from the plurality of data blocks based on the observed accesses;
determining storage sizes for the one or more data blocks from the plurality of data blocks;
sorting the one or more data blocks based at least in part on the determined access patterns and on the determined storage sizes, wherein the sorting comprising:
assigning the one or more data blocks to a plurality of buckets each bucket associated with a particular access pattern level and data block storage size requirement, wherein assigning the one or more data blocks to the plurality of buckets comprises:
matching a determined access pattern and a determined storage size of a particular data block to an access pattern level and a data block storage size requirement of a particular bucket from the plurality of buckets; and
assigning the particular data block to the particular bucket based on the matching; and
redistributing the one or more data blocks across a plurality of storage devices of the distributed storage based on the sorting of the one or more data blocks, wherein the redistributing comprises:
determining a total number of data blocks assigned to a particular bucket from the plurality of buckets;
calculating a target number of data blocks for each of the plurality of storage devices for the particular bucket by dividing the determined total number of data blocks by a number of the plurality of storage devices; and
redistributing the data blocks assigned to the particular bucket across the plurality of storage devices based on the calculated target number of data blocks for each of the plurality of storage devices for the particular bucket.
2. The computer-implemented method of claim 1 , wherein redistributing the one or more data blocks across the plurality of storage devices based on the sorting comprises uniformly redistributing data blocks having similar access patterns and storage sizes across the plurality of storage devices.
3. The computer-implemented method of claim 1 , wherein redistributing the data blocks across the plurality of storage devices comprises:
selecting a bucket from the plurality of buckets, the bucket having an access pattern level specifying an access time that is more recent than an access time specified by an access pattern level for another bucket from the plurality of data buckets; and
redistributing data blocks assigned to the selected bucket prior to redistributing data blocks assigned to the another bucket.
4. A non-transitory computer readable storage medium executing computer program instructions for storing data based on access patterns, the computer program instructions comprising instructions for:
observing accesses to the plurality of data blocks stored in the distributed storage; and
redistributing the plurality of data blocks in the distributed storage, wherein redistribution of the plurality of data blocks in the distributed storage comprises:
determining access patterns for one or more data blocks from the plurality of data blocks based on the observed accesses;
determining storage sizes for the one or more data blocks from the plurality of data blocks;
sorting the one or more data blocks based at least in part on the determined access patterns and on the determined storage sizes, wherein the sorting comprising:
assigning the one or more data blocks to a plurality of buckets each bucket associated with a particular access pattern level and data block storage size requirement, wherein assigning the one or more data blocks to the plurality of buckets comprises:
matching a determined access pattern and a determined storage size of a particular data block to an access pattern level and a data block storage size requirement of a particular bucket from the plurality of buckets; and
assigning the particular data block to the particular bucket based on the matching; and
redistributing the one or more data blocks across a plurality of storage devices of the distributed storage based on the sorting of the one or more data blocks, wherein the redistributing comprises:
determining a total number of data blocks assigned to a particular bucket from the plurality of buckets;
calculating a target number of data blocks for each of the plurality of storage devices for the particular bucket by dividing the determined total number of data blocks by a number of the plurality of storage devices; and
redistributing the data blocks assigned to the particular bucket across the plurality of storage devices based on the calculated target number of data blocks for each of the plurality of storage devices for the particular bucket.
5. The medium of claim 4 , wherein redistributing the one or more data blocks across the plurality of storage devices based on the sorting comprises uniformly redistributing data blocks having similar access patterns and storage sizes across the plurality of storage devices.
6. The medium of claim 4 , wherein redistributing the data blocks across the plurality of storage devices comprises:
selecting a bucket from the plurality of buckets, the bucket having an access pattern level specifying an access time that is more recent than an access time specified by an access pattern level for another bucket from the plurality of data buckets; and
redistributing data blocks assigned to the selected bucket prior to redistributing data blocks assigned to the another bucket.
7. A system comprising:
a non-transitory computer readable storage medium storing processor-executable computer program instructions for redistributing data, the instructions comprising instructions for:
observing accesses to the plurality of data blocks stored in the distributed storage; and
redistributing the plurality of data blocks in the distributed storage, wherein redistribution of the plurality of data blocks in the distributed storage comprises:
determining access patterns for one or more data blocks from the plurality of data blocks based on the observed accesses;
determining storage sizes for the one or more data blocks from the plurality of data blocks;
sorting the one or more data blocks based at least in part on the determined access patterns and on the determined storage sizes, wherein the sorting comprising:
assigning the one or more data blocks to a plurality of buckets each bucket associated with a particular access pattern level and data block storage size requirement, wherein assigning the one or more data blocks to the plurality of buckets comprises:
matching a determined access pattern and a determined storage size of a particular data block to an access pattern level and a data block storage size requirement of a particular bucket from the plurality of buckets; and
assigning the particular data block to the particular bucket based on the matching; and
redistributing the one or more data blocks across a plurality of storage devices of the distributed storage based on the sorting of the one or more data blocks, wherein the redistributing comprises:
determining a total number of data blocks assigned to a particular bucket from the plurality of buckets;
calculating a target number of data blocks for each of the plurality of storage devices for the particular bucket by dividing the determined total number of data blocks by a number of the plurality of storage devices; and
redistributing the data blocks assigned to the particular bucket across the plurality of storage devices based on the calculated target number of data blocks for each of the plurality of storage devices for the particular bucket; and
a processor for executing the computer program instructions.
8. The system of claim 7 , wherein redistributing the one or more data blocks across the plurality of storage devices based on the sorting comprises uniformly redistributing data blocks having similar access patterns and storage sizes across the plurality of storage devices.
9. The system of claim 7 , wherein redistributing the data blocks across the plurality of storage devices comprises:
selecting a bucket from the plurality of buckets, the bucket having an access pattern level specifying an access time that is more recent than an access time specified by an access pattern level for another bucket from the plurality of data buckets; and
redistributing data blocks assigned to the selected bucket prior to redistributing data blocks assigned to the another bucket.