Data block migration
Techniques and mechanisms are provided for migrating data blocks around a cluster during node addition and node deletion. Migration requires no downtime, as a newly added node is immediately operational while the data blocks are being moved. Blockmap files and deduplication dictionaries need not be updated.
1. A method, comprising:
receiving a request to add a new node from a data storage cluster, the data storage cluster maintaining a plurality of deduplicated data segments in a plurality of suitcases at particular nodes in the data storage cluster, wherein the plurality of suitcases include datastore suitcases created after optimizing a file, each datastore suitcase comprising a data structure including deduplicated data segments, index information, offset information, data reference count information, and last file reference information, wherein optimizing a file includes compressing the file;
generating a plurality of new keys associated with a mapping function, the mapping function using a particular key to identify a particular node containing a particular suitcase, wherein the plurality of new keys are used to identify particular suitcases stored in particular nodes, including the new node, of the data storage cluster,
copying data including suitcases and their corresponding deduplicated data segments from the plurality of existing nodes to the new node, in accordance with the mapping function and new keys, to rebalance data across the data storage cluster,
wherein performing data access after data migration includes accessing a stub file corresponding to a virtual image of the optimized file, the stub file providing a suitcase identifier that specifies a node.
2. The method of claim 1 , wherein each of the plurality of new keys includes a node number.
3. The method of claim 1 , wherein the plurality of new keys correspond to a plurality of blockmap files, but the blockmap files do not contain references to the new-keys.
4. The method of claim 3 , wherein each blockmap file includes offset, length, and location identifiers for identifying segments in a plurality of suitcases.
5. The method of claim 3 , wherein the plurality of blockmap files remain unchanged after adding the new node.
6. The method of claim 1 , wherein the plurality of new keys are a plurality of suitcase identifiers (scids).
7. The method of claim 6 , wherein the mapping function comprises the following:
#define get_the_node_number_from_the_scid(_scid_) scid_to_node_array [_scid_% MAX_CLUSTER_SIZE]
wherein scid represents a suitcase ID and wherein MAX_CLUSTER_SIZE represents a maximum cluster size of the data storage cluster.
8. A system, comprising:
a processor; and
memory comprising instructions to execute a method, the method comprising:
receiving a request to add a new node from a data storage cluster, the data storage cluster maintaining a plurality of deduplicated data segments in a plurality of suitcases at particular nodes in the data storage cluster, wherein the plurality of suitcases include datastore suitcases created after optimizing a file, each datastore suitcase comprising a data structure including deduplicated data segments, index information, offset information, data reference count information, and last file reference information, wherein optimizing a file includes compressing the file;
generating a plurality of new keys associated with a mapping function, the mapping function using a particular key to identify a particular node containing a particular suitcase, wherein the plurality of new keys are used to identify particular suitcases stored in particular nodes, including the new node, of the data storage cluster,
copying data including suitcases and their corresponding deduplicated data segments from the plurality of existing nodes to the new node, in accordance with the mapping function and new keys, to rebalance data across the data storage cluster,
wherein performing data access after data migration includes accessing a stub file corresponding to a virtual image of the optimized file, the stub file providing a suitcase identifier that specifies a node.
9. The system of claim 8 , wherein each of the plurality of new keys includes a node number.
10. The system of claim 8 , wherein the plurality of new keys correspond to a plurality of blockmap files, but the blockmap files do not contain references to the new keys.
11. The system of claim 10 , wherein each blockmap file includes offset, length, and location identifiers for identifying segments in a plurality of suitcases.
12. The system of claim 10 , wherein the plurality of blockmap files remain unchanged after adding the new node.
13. The system of claim 8 , wherein the plurality of new keys are a plurality of suitcase identifiers (scids).
14. The system of claim 13 , wherein the mapping function comprises the following:
#define get_the_node_number_from_the_scid(_scid_) scid_to_node_array [_scid_% MAX_CLUSTER_SIZE]
wherein scid represents a suitcase ID and wherein MAX CLUSTER SIZE represents a maximum cluster size of the data storage cluster.
15. A non-transitory computer readable medium comprising computer code for:
receiving a request to add a new node from a data storage cluster, the data storage cluster maintaining a plurality of deduplicated data segments in a plurality of suitcases at particular nodes in the data storage cluster, wherein the plurality of suitcases include datastore suitcases created after optimizing a file, each datastore suitcase comprising a data structure including deduplicated data segments, index information, offset information, data reference count information, and last file reference information, wherein optimizing a file includes compressing the file;
generating a plurality of new keys associated with a mapping function, the mapping function using a particular key to identify a particular node containing a particular suitcase, wherein the plurality of new keys are used to identify particular suitcases stored in particular nodes, including the new node, of the data storage cluster;
copying data including suitcases and their corresponding deduplicated data segments from the plurality of existing nodes to the new node, in accordance with the mapping function and new keys, to rebalance data across the data storage cluster,
wherein performing data access after data migration includes accessing a stub file corresponding to a virtual image of the optimized file, the stub file providing a suitcase identifier that specifies a node.
16. The non-transitory computer readable medium of claim 15 , wherein each of the plurality of new keys includes a node number.
17. The non-transitory computer readable medium of claim 15 , wherein the plurality of new keys correspond to a plurality of blockmap files, but the blockmap files do not contain references to the new keys.
18. The non-transitory computer readable medium of claim 17 , wherein each blockmap file includes offset, length, and location identifiers for identifying segments in a plurality of suitcases.
19. The non-transitory computer readable medium of claim 17 , wherein the plurality of blockmap files remain unchanged after adding the new node.
20. The non-transitory computer readable medium of claim 15 , wherein the plurality of new keys are a plurality of suitcase identifiers (scids).