Data block migration
View Patent ↗Techniques and mechanisms are provided for migrating data blocks around a cluster during node addition and node deletion. Migration requires no downtime, as a newly added node is immediately operational while the data blocks are being moved. Blockmap files and deduplication dictionaries need not be updated.
1. A method, comprising:
receiving a request to add a new node from a data storage cluster, the data storage cluster maintaining a plurality of deduplicated data segments in a plurality of suitcases at particular nodes in the data storage cluster, wherein a plurality of blockmap files include information for locating which suitcases in the plurality of suitcases contain particular deduplicated data segments, wherein the plurality of suitcases include datastore suitcases created after optimizing a file, each datastore suitcase comprising a data structure including deduplicated data segments, index information, offset information, data reference count information, and last file reference information, wherein optimizing a file includes compressing the file;
generating a plurality of new keys associated with a mapping function separate from the plurality of blockmap files, the mapping function using a particular key to identify a particular node containing a particular suitcase, wherein the plurality of new keys are used to identify particular suitcases stored in particular nodes, including the new node, of the data storage cluster, wherein the plurality of blockmap files, being separate from the mapping function, do not contain references to the new keys;
copying data including suitcases and their corresponding deduplicated data segments from the plurality of existing nodes to the new node, in accordance with the mapping function and new keys, to rebalance data across the data storage cluster,
wherein performing data access after data migration includes accessing a stub file corresponding to a virtual image of the optimized file, the stub file providing a suitcase identifier that specifies a node.
2. The method of claim 1 , wherein each of the plurality of new keys includes a node number.
3. The method of claim 1 ; wherein the plurality of new keys correspond to the plurality of blockmap files, but the blockmap files still do not contain references to the new keys.
4. The method of claim 3 , wherein each blockmap file includes offset, length, and location identifiers for identifying segments in a plurality of suitcases.
5. The method of claim 3 , wherein the plurality of blockmap files remain unchanged after adding the new node.
6. The method of claim 1 , wherein the plurality of new keys are a plurality of suitcase identifiers (scids).
7. The method of claim 6 , wherein the mapping function comprises the following:
#define get_the_node_number_from_the_scid(_scid_)
scid_to_node_array[_scid_% MAX_CLUSTER_SIZE]
wherein scid represents a suitcase ID and wherein MAX_CLUSTER_SIZE represents a maximum cluster size of the data storage cluster.
8. The wherein the new node is a storage device.
9. The method of claim 1 , wherein the new node is a storage array.
10. The method of claim 1 , wherein a plurality of segments are copied onto a suitcase maintained at the new node.
11. The method of claim 1 , wherein the mapping function can be changed while the plurality of new keys are being generated.
12. A system, comprising:
an interface configured to receive a request to add a new node from a data storage cluster, the data storage cluster maintaining a plurality of deduplicated data segments in a plurality of suitcases at particular nodes in the data storage cluster, wherein a plurality of blockmap files include information for locating which suitcases in the plurality of suitcases contain particular deduplicated data segments, wherein the plurality of suitcases include datastore suitcases created after optimizing a file, each datastore suitcase comprising a data structure including deduplicated data segments, index information, offset information, data reference count information, and last file reference information, wherein optimizing a file includes compressing the file;
a processor configured to generate a plurality of new keys associated with a mapping function separate from the plurality of blockmap files, the mapping function using a particular key to identify a particular node containing a particular suitcase, wherein the plurality of new keys are used to identify particular suitcases stored in particular nodes, including the new node, of the data storage cluster, wherein the plurality of blockmap files, being separate from the mapping function, do not contain references to the new keys, the processor further configured to copy data including suitcases and their corresponding deduplicated data segments from the plurality of existing nodes to the new node, in accordance with the mapping function and new keys, to rebalance data across the data storage cluster,
wherein the processor is further configured to perform data access after data migration, wherein performing data access includes accessing a stub file corresponding to a virtual image of the optimized file, the stub file providing a suitcase identifier that specifies a node.
13. The system claim 12 , wherein each of the plurality of new keys includes a node number.
14. The system of claim 12 , wherein the plurality of new keys correspond to the plurality of blockmap files, but the blockmap files still do not contain references to the new keys.
15. The system of claim 14 ; wherein each blockmap file includes offset, length, and location identifiers for identifying segments in a plurality of suitcases.
16. The system of claim 14 , wherein the plurality of blockmap files remain unchanged after adding the new node.
17. The system of claim 12 , wherein the plurality of new keys are a plurality of suitcase identifiers (scids).
18. A non-transitory computer readable medium comprising computer code for:
receiving a request to add a new node from a data storage cluster, the data storage cluster maintaining a plurality of deduplicated data segments in a plurality of suitcases at particular nodes in the data storage cluster, wherein a plurality of blockmap files include information for locating which suitcases in the plurality of suitcases contain particular deduplicated data segments, wherein the plurality of suitcases include datastore suitcases created after optimizing a file, each datastore suitcase comprising a data structure including deduplicated data segments, index information, offset information, data reference count information, last file reference information, wherein optimizing a file includes compressing the file;
generating a plurality of new keys associated with a mapping function separate from the plurality of blockmap files, the mapping function using a particular key to identify a particular node containing a particular suitcase, wherein the plurality of new keys are used to identify particular suitcases stored in particular nodes, including the new node, of the data storage cluster, wherein the plurality of blockmap files, being separate from the mapping function, do not contain references to the new keys;
copying data including suitcases and their corresponding deduplicated data segments from the plurality of existing nodes to the new node, in accordance with the mapping function and new keys, to rebalance data across the data storage cluster,
wherein performing data access after data migration includes accessing a stub file corresponding to a virtual image of the optimized file, the stub file providing a suitcase identifier that specifies a node.
19. The computer readable medium of claim 18 , wherein each of the plurality of new keys includes a node number.
20. The computer readable medium of claim 18 , wherein the plurality of new keys correspond to the plurality of blockmap files, but the blockmap files still do not contain references to the new keys.