Capacity based load balancing in distributed storage systems with deduplication and compression functionalities

1. A method for migrating a first group of data blocks stored in a first node to a second node, comprising:

determining, for each of a plurality of groups of data blocks stored in the first node, a corresponding first metric representing a total size for storing all data blocks in the corresponding group without deduplication, wherein each data block of the plurality of groups of data blocks is associated with a corresponding physical address;

determining, for each of the plurality of groups of data blocks stored in the first node, a corresponding second metric representing a total size for storing all unique data blocks in the corresponding group, wherein a given data block is a unique data block if no other data block of the plurality of groups of data blocks is associated with the same physical address as the given data block;

selecting the first group of data blocks for migration from the first node to the second node based on the corresponding first metric satisfying a first criteria and the corresponding second metric satisfying a second criteria; and

migrating the first group of data blocks to the second node.

2. The method of claim 1 , wherein the second criteria is that the corresponding second metric is equal to or greater than a median of all second metrics of the plurality of groups of data blocks.

3. The method of claim 1 , wherein the first criteria is that the corresponding first metric has a minimum value among all first metrics of all of the plurality of groups of data blocks associated with a second metric that satisfies the second criteria.

4. The method of claim 1 , further comprising:

determining, for each of the plurality of groups of data blocks stored in the first node, a corresponding third metric representing a total size for storing all distinct data blocks in the corresponding group, wherein the total size for storing all distinct data blocks in the corresponding group comprises a total size for storing all data blocks in the corresponding group with deduplication across the corresponding group; and

selecting the second node from among a plurality of nodes based on the second node having available storage capacity equal to or greater than the third metric of the first group of data blocks.

5. The method of claim 4 , wherein multiple nodes of the plurality of nodes have available storage capacity equal to or greater than the third metric of the first group of data blocks, and wherein the second node is randomly selected from among the multiple of the plurality of nodes.

6. The method of claim 1 , wherein the first metric is determined based on uncompressed sizes of data blocks, and wherein the second metric is determined based on compressed sizes of data blocks.

7. The method of claim 1 , wherein each of the plurality of groups of data blocks is a component object of a composite object.

8. A non-transitory computer readable storage medium comprising instructions that are executable in a computer system to cause the computer system to perform a method for migrating a first group of data blocks stored in a first node to a second node, comprising:

determining, for each of a plurality of groups of data blocks stored in the first node, a corresponding first metric representing a total size for storing all data blocks in the corresponding group without deduplication, wherein each data block of the plurality of groups of data blocks is associated with a corresponding physical address;

determining, for each of the plurality of groups of data blocks stored in the first node, a corresponding second metric representing a total size for storing all unique data blocks in the corresponding group, wherein a given data block is a unique data block if no other data block of the plurality of groups of data blocks is associated with the same physical address as the given data block;

selecting the first group of data blocks for migration from the first node to the second node based on the corresponding first metric satisfying a first criteria and the corresponding second metric satisfying a second criteria; and

migrating the first group of data blocks to the second node.

9. The non-transitory computer readable storage medium of claim 8 , wherein the second criteria is that the corresponding second metric is equal to or greater than a median of all second metrics of the plurality of groups of data blocks.

10. The non-transitory computer readable storage medium of claim 8 , wherein the first criteria is that the corresponding first metric has a minimum value among all first metrics of all of the plurality of groups of data blocks associated with a second metric that satisfies the second criteria.

11. The non-transitory computer readable storage medium of claim 8 , wherein the method further comprises:

determining, for each of the plurality of groups of data blocks stored in the first node, a corresponding third metric representing a total size for storing all distinct data blocks in the corresponding group, wherein the total size for storing all distinct data blocks in the corresponding group comprises a total size for storing all data blocks in the corresponding group with deduplication across the corresponding group; and

selecting the second node from among a plurality of nodes based on the second node having available storage capacity equal to or greater than the third metric of the first group of data blocks.

12. The non-transitory computer readable storage medium of claim 11 , wherein multiple nodes of the plurality of nodes have available storage capacity equal to or greater than the third metric of the first group of data blocks, and wherein the second node is randomly selected from among the multiple of the plurality of nodes.

13. The non-transitory computer readable storage medium of claim 8 , wherein the first metric is determined based on uncompressed sizes of data blocks, and wherein the second metric is determined based on compressed sizes of data blocks.

14. The non-transitory computer readable storage medium of claim 8 , wherein each of the plurality of groups of data blocks is a component object of a composite object.

15. A computer system comprising:

local storage of a first node that stores a plurality of groups of data blocks; and

a processor configured to:

determine, for each of the plurality of groups of data blocks stored in the first node, a corresponding first metric representing a total size for storing all data blocks in the corresponding group without deduplication, wherein each data block of the plurality of groups of data blocks is associated with a corresponding physical address;

determine, for each of the plurality of groups of data blocks stored in the first node, a corresponding second metric representing a total size for storing all unique data blocks in the corresponding group, wherein a given data block is a unique data block if no other data block of the plurality of groups of data blocks is associated with the same physical address as the given data block;

select the first group of data blocks for migration from the first node to the second node based on the corresponding first metric satisfying a first criteria and the corresponding second metric satisfying a second criteria; and

migrate the first group of data blocks to the second node.

16. The computer system of claim 15 , wherein the second criteria is that the corresponding second metric is equal to or greater than a median of all second metrics of the plurality of groups of data blocks.

17. The computer system of claim 15 , wherein the first criteria is that the corresponding first metric has a minimum value among all first metrics of all of the plurality of groups of data blocks associated with a second metric that satisfies the second criteria.

18. The computer system of claim 15 , wherein the processor is further configured to:

determine, for each of the plurality of groups of data blocks stored in the first node, a corresponding third metric representing a total size for storing all distinct data blocks in the corresponding group, wherein the total size for storing all distinct data blocks in the corresponding group comprises a total size for storing all data blocks in the corresponding group with deduplication across the corresponding group; and

select the second node from among a plurality of nodes based on the second node having available storage capacity equal to or greater than the third metric of the first group of data blocks.

19. The computer system of claim 18 , wherein multiple nodes of the plurality of nodes have available storage capacity equal to or greater than the third metric of the first group of data blocks, and wherein the second node is randomly selected from among the multiple of the plurality of nodes.

20. The computer system of claim 15 , wherein the first metric is determined based on uncompressed sizes of data blocks, and wherein the second metric is determined based on compressed sizes of data blocks.