Dynamic physical capacity allocation in an unbalanced CAS system
An aspect of dynamic physical capacity allocation in an unbalanced content-addressable storage system includes initiating a scale up for a storage cluster. An aspect further includes determining a target number of hash-to-physical-location-on-disk (h2d) slices to be assigned to each data module (D module), and reassigning h2d slices assigned to D modules having greater than the target number of h2d slices to D modules having less h2d slices than the target number.
1. A method, comprising:
storing a hash-to-physical-location-on-disk (h2d) table in persistent storage of a storage cluster, the h2d table configured to enable routing of data with different hashes to corresponding data modules (D modules) and storing assignments of h2d slices to corresponding D modules, the corresponding D modules configured to read and write data to and from storage devices for corresponding assigned h2d slices, wherein a h2d slice signifies a part of a hash;
initiating a scale up for the storage cluster, the scale up including adding storage drives to one or more bricks of the storage cluster, wherein the storage cluster is determined to be unbalanced when physical capacity differs between the bricks;
determining, by a management module of the storage cluster responsive to identifying the storage cluster as unbalanced, a target number of the h2d slices to be assigned to each of the D modules, wherein the target number of h2d slices to be assigned to each D module of the D modules is determined by:
determining a total number of the h2d slices to assign to the D modules;
determining a total physical capacity of the storage cluster owned by the D modules; and
dividing the total number of the h2d slices among the D modules proportionally based on the proportion of the total physical capacity owned by each of the D modules, wherein at least a first proportion owned by a first D module differs from a second proportion owned by a second D module, such that, upon the dividing, each D module of the D modules owns a number of the h2d slices of the total number of h2d slices in a same proportion as a ratio of the total physical capacity owned by each D module, wherein responsive to the dividing, at least one of the D modules owns a different number of the h2d slices;
identifying a remainder number of the h2d slices resulting from the dividing; and
reassigning, by the management module in the h2d table and without accessing the D modules, a portion of the h2d slices assigned to one or more of the D modules identified as having greater than the target number of h2d slices to another one or more of the D modules identified as having fewer h2d slices than the target number, wherein the reassigning includes reassigning, one-by-one, the remainder number of h2d slices in sequence to the D modules.
2. The method of claim 1 , wherein the h2d table including updated assignments of the h2d slices is retrieved for use by the management module in response to a next scale up.
3. The method of claim 2 , wherein the scale up is performed in a content addressable system.
4. The method of claim 1 , wherein configuration of the storage cluster is determined to be unbalanced when the D modules are responsible for unequal portions of physical capacity of the storage devices.
5. The method of claim 1 , wherein each of the D modules is responsible for an amount of data proportional to the number of assigned h2d slices.
6. A system, comprising:
a memory comprising computer-executable instructions; and
a processor executing the computer-executable instructions, the computer-executable instructions when executed by the processor cause the processor to perform operations comprising:
storing a hash-to-physical-location-on-disk (h2d) table in persistent storage of a storage cluster, the h2d table configured to enable routing of data with different hashes to corresponding data modules (D modules) and storing assignments of h2d slices to corresponding D modules, the corresponding D modules configured to read and write data to and from one or more storage devices for corresponding assigned h2d slices, wherein a h2d slice signifies a part of a hash;
initiating a scale up for the storage cluster, the scale up including adding storage drives to one or more bricks of the storage cluster, wherein the storage cluster is determined to be unbalanced when physical capacity differs between the bricks;
determining, by a management module of the storage cluster responsive to identifying the storage cluster as unbalanced, a target number of the h2d slices to be assigned to each of the D modules, wherein the target number of h2d slices to be assigned to each D module of the D modules is determined by:
determining a total number of the h2d slices to assign to the D modules;
determining a total physical capacity of the storage cluster owned by the D modules; and
dividing the total number of the h2d slices among the D modules proportionally based on the proportion of the total physical capacity owned by each of the D modules, wherein at least a first proportion owned by a first D module differs from a second proportion owned by a second D module, such that, upon the dividing, each D module of the D modules owns a number of the h2d slices of the total number of h2d slices in a same proportion as a ratio of the total physical capacity owned by each D module, wherein responsive to the dividing, at least one of the D modules owns a different number of the h2d slices;
identifying a remainder number of the h2d slices resulting from the dividing; and
reassigning, by the management module in the h2d table and without accessing the D modules, a portion of the h2d slices assigned to one or more of the D modules identified as having greater than the target number of h2d slices to another one or more of the D modules identified as having fewer h2d slices than the target number, wherein the reassigning includes reassigning, one-by-one, the remainder number of h2d slices in sequence to the D modules.
7. The system of claim 6 , wherein the h2d table including updated assignments of the h2d slices is retrieved for use by the management module in response to a next scale up.
8. The system of claim 7 , wherein the scale up is performed in a content addressable system.
9. The system of claim 6 , wherein configuration of the storage cluster is determined to be unbalanced when the D modules are responsible for unequal portions of physical capacity of the storage devices.
10. The system of claim 6 , wherein each of the D modules is responsible for an amount of data proportional to the number of assigned h2d slices.
11. A computer program product embodied on a non-transitory computer readable medium, the computer program product including instructions that, when executed by a computer causes the computer to perform operations comprising:
storing a hash-to-physical-location-on-disk (h2d) table in persistent storage of a storage cluster, the h2d table configured to enable routing of data with different hashes to corresponding data modules (D modules) and storing assignments of h2d slices to corresponding D modules, the corresponding D modules configured to read and write data to and from one or more storage devices for corresponding assigned h2d slices, wherein a h2d slice signifies a part of a hash;
initiating a scale up for the storage cluster, the scale up including adding storage drives to one or more bricks of the storage cluster, wherein the storage cluster is determined to be unbalanced when physical capacity differs between the bricks;
determining, by a management module of the storage cluster responsive to identifying the storage cluster as unbalanced, a target number of the h2d slices to be assigned to each of the D modules, wherein the target number of h2d slices to be assigned to each D module of the D modules is determined by:
determining a total number of the h2d slices to assign to the D modules;
determining a total physical capacity of the storage cluster owned by the D modules; and
dividing the total number of the h2d slices among the D modules proportionally based on the proportion of the total physical capacity owned by each of the D modules, wherein at least a first proportion owned by a first D module differs from a second proportion owned by a second D module, such that, upon the dividing, each D module of the D modules owns a number of the h2d slices of the total number of h2d slices in a same proportion as a ratio of the total physical capacity owned by each D module, wherein responsive to the dividing, at least one of the D modules owns a different number of the h2d slices;
identifying a remainder number of the h2d slices resulting from the dividing; and
reassigning, by the management module in the h2d table and without accessing the D modules, a portion of the h2d slices assigned to one or more of the D modules identified as having greater than the target number of h2d slices to another one or more of the D modules identified as having fewer h2d slices than the target number, wherein the reassigning includes reassigning, one-by-one, the remainder number of h2d slices in sequence to the D modules.
12. The computer program product of claim 11 , wherein the h2d table including updated assignments of the h2d slices is retrieved for use by the management module in response to a next scale up.
13. The computer program product of claim 11 , wherein the scale up is performed in a content addressable system.
14. The computer program product of claim 11 , configuration of the storage cluster is determined to be unbalanced when the D modules are responsible for unequal portions of physical capacity of the storage devices.
15. The computer program product of claim 11 , wherein each of the D modules is responsible for an amount of data proportional to the number of assigned h2d slices.