Storage optimization manager
Techniques and mechanisms provide a storage optimization manager. Data may be optimized and maintained on various nodes in a cluster. Particular nodes may be overburdened while other nodes remain relatively unused. Techniques are provided to efficiently optimize data onto nodes to enhance operational efficiency. Data access requests for optimized data are monitored and managed to allow for intelligent maintenance of optimized data.
1. A method, comprising:
monitoring data access requests for optimized data maintained on a plurality of nodes in a cluster, wherein the optimized data comprises deduplicated data;
detecting a optimized data storage imbalance;
generating a plurality of new keys associated with a mapping function, the mapping function identifying a particular node corresponding to a particular key, wherein each of the plurality of new keys includes a node number, wherein the plurality of new keys correspond to a plurality of blockmap files, wherein each blockmap file includes offset, length, and location identifiers for identifying segments in a plurality of suitcases, each suitcase identifying the location of the segments on disk, wherein the plurality of blockmap files remain unchanged after redistributing optimized data;
redistributing optimized data across the plurality of nodes in the cluster, wherein redistributing the optimized data involves updating one or more of the suitcases to modify one or more of the locations of the segments on disk.
2. The method of claim 1 , wherein the plurality of new keys are a plurality of suitcase identifiers (scids).
3. The method of claim 2 , wherein the mapping function is used to identify a particular node and comprises the following:
#define get_the_node_number_from_the_scid(_scid_)
scid_to_node_array[_scid — % MAX_CLUSTER_SIZE].
4. The method of claim 1 , wherein the data imbalance is a disproportionate rate of data access at a particular node.
5. The method of claim 1 , wherein the data imbalance corresponds to overuse of particular nodes and underuse of other nodes.
6. The method of claim 1 , wherein a plurality of segments are redistributed across the plurality of nodes.
7. The method of claim 1 , wherein the mapping function can be changed while the plurality of new keys are being generated.
8. A system, comprising:
an interface configured to monitor data access requests for optimized data maintained on a plurality of nodes in a cluster, wherein the optimized data comprises deduplicated data;
a processor configured to detect an optimized data storage imbalance and generate a plurality of new keys associated with a mapping function, the mapping function identifying a particular node corresponding to a particular key, wherein each of the plurality of new keys includes a node number, wherein the plurality of new keys correspond to a plurality of blockmap files, wherein each blockmap file includes offset, length, and location identifiers for identifying segments in a plurality of suitcases, wherein the plurality of blockmap files remain unchanged after redistributing optimized data;
wherein optimized data is redistributed across the plurality of nodes in the cluster, wherein redistributing the optimized data involves updating one or more of the suitcases to modify one or more of the locations of the segments on disk.
9. The system of claim 8 , wherein the plurality of new keys are a plurality of suitcase identifiers (scids).
10. The method of claim 9 , wherein the mapping function is used to identify a particular node and comprises the following:
#define get_the_node_number_from_the_scid(_scid_)
scid_to_node_array[_scid_% MAX_CLUSTER_SIZE].
11. The system of claim 8 , wherein the data imbalance is a disproportionate rate of data access at a particular node.
12. The system of claim 8 , wherein the data imbalance corresponds to overuse of particular nodes and underuse of other nodes.
13. An apparatus, comprising:
means for monitoring data access requests for optimized data maintained on a plurality of nodes in a cluster, wherein the optimized data comprises deduplicated data;
means for detecting a optimized data storage imbalance;
means for generating a plurality of new keys associated with a mapping function, the mapping function identifying a particular node corresponding to a particular key, wherein each of the plurality of new keys includes a node number, wherein the plurality of new keys correspond to a plurality of blockmap files, wherein each blockmap file includes offset, length, and location identifiers for identifying segments in a plurality of suitcases, wherein the plurality of blockmap files remain unchanged after redistributing optimized data;
means for redistributing optimized data across the plurality of nodes in the cluster, wherein redistributing the optimized data involves updating one or more of the suitcases to modify one or more of the locations of the segments on disk.