Sub-cluster recovery using a partition group index
Embodiments disclosed herein provide systems, methods, and computer readable media for sub-cluster recovery in a data storage environment having a plurality of storage nodes. In a particular embodiment, the method provides scanning data items in the plurality of nodes. While scanning, the method further provides indexing the data items into an index of a plurality of partition groups. Each partition group includes data items owned by a particular one of the plurality of storage nodes. The method then provides storing the index.
1. A method for sub-cluster recovery in a data storage environment having a plurality of nodes, the method comprising:
scanning data items stored in a first plurality of nodes of a first cluster;
while scanning, creating a partition group index indexing the data items into a plurality of partition groups, wherein each partition group corresponds to a node of the first plurality of nodes and comprises a subset of the data items stored in the node;
storing the partition group index;
instantiating a second cluster using a second plurality of nodes;
identifying the data items included in the plurality of partition groups according to the partition group index; and
loading the data items included in the plurality of partition groups onto the second plurality of nodes.
2. The method of claim 1 , further comprising:
recovering each of the first plurality of nodes by:
identifying the data items stored in the node based on mappings between the plurality of partition groups and the first plurality of nodes, and
loading the identified data items onto the node.
3. The method of claim 1 , wherein the instantiating the second cluster comprises:
for each node of the second plurality of nodes: generating per node data based on mappings between the plurality of partition groups and the first plurality of nodes of the first cluster.
4. The method of claim 1 , wherein the second plurality of nodes include less nodes than the first plurality of nodes and wherein instantiating the second cluster comprises further:
generating a partition group to node mapping for the second cluster from the partition group index.
5. The method of claim 1 , further comprising:
while scanning the data items, identifying duplicate data items in the first cluster;
deduplicating the duplicate data items;
repackaging each of the duplicate data items into respective deduplicated data units; and
storing the deduplicated data units to a secondary data repository.
6. The method of claim 5 ; further comprising:
determining a degree of duplicates of the data items in the first cluster; and
comparing the degree of duplicates to a predetermined level of consistency,
wherein the deduplication of the duplicate data items is performed responsive to determining the degree of duplicates is greater than the predetermined level of consistency.
7. The method of claim 5 , wherein the plurality of partition groups include deduplicated data items.
8. The method of claim 5 ; wherein storing the deduplicated data units comprises storing a data version of the data items, further comprising:
compiling the deduplicated data units into the data version of the data items.
9. The method of claim 1 , wherein the data items are stored in a NoSQL data store.
10. A system for sub-cluster recovery in a data storage environment having a plurality of nodes, the system comprising:
one or more non-transitory computer readable storage media;
a processing system operatively coupled with the one or more non-transitory computer readable storage media; and
program instructions stored on the one or more non-transitory computer readable storage media that, when read and executed by the processing system, direct the processing system to:
scan data items stored in a first plurality of nodes of a first cluster;
while scanning, creating a partition group index indexing the data items into a plurality of partition groups, wherein each partition group corresponds to a node of the first plurality of nodes and comprises a subset of the data items stored in the node;
store the partition group index;
instantiate a second cluster using a second plurality of nodes;
identifying the data items included in the plurality of partition groups according to the partition group index; and
load the data items included in the plurality of partition groups onto the second plurality of nodes.
11. The system of claim 10 , wherein the program instructions further direct the processing system to:
recover each of the plurality of nodes by:
identifying the data items stored in the node based on mappings between the plurality of partition groups and the first plurality of nodes, and
loading the identified data items onto the node.
12. The system of claim 10 , wherein the program instructions configured to direct the processing system to instantiate the second cluster further direct the processing system to:
for each node of the second plurality of nodes: generate per node data based on mappings between the plurality of partition groups and the first plurality of nodes of the first cluster.
13. The system of claim 10 , wherein the second plurality of nodes include less nodes than the first plurality of nodes and wherein the program instructions configured to direct the processing system to instantiate the second cluster directs the processing system to:
generate a partition group to node mapping for the second cluster from the partition group index.
14. The system of claim 10 , wherein the program instructions further direct the processing system to:
while scanning the data items, identify duplicate data items in the first cluster;
deduplicate the duplicate data items;
repackage each of the duplicate data items into respective deduplicated data units; and
store the deduplicated data units to a secondary data repository.
15. The system of claim 14 , wherein the program instructions further direct the processing system to:
determine a degree of duplicates of the data items in the first cluster; and
compare the degree of duplicates to a predetermined level of consistency,
wherein the deduplication of the duplicate data items is performed responsive to determining the degree of duplicates is greater than the predetermined level of consistency.
16. The system of claim 14 , wherein the plurality of partition groups include deduplicated data items.
17. The system of claim 14 , wherein storing the deduplicated data units comprises storing a data version of the data items, and wherein the program instructions further direct the processing system to:
compile the deduplicated data units into the data version of the data items.
18. The system of claim 10 , wherein the data items are stored in a NoSQL data store.