Sub-cluster recovery using a partition group index
Embodiments disclosed herein provide systems, methods, and computer readable media for sub-cluster recovery in a data storage environment having a plurality of storage nodes. In a particular embodiment, the method provides scanning data items in the plurality of nodes. While scanning, the method further provides indexing the data items into an index of a plurality of partition groups. Each partition group includes data items owned by a particular one of the plurality of storage nodes. The method then provides storing the index.
1. A method for sub-cluster recovery in a data storage environment having a plurality of nodes, the method comprising:
creating a partition group index that indexes a set of data items into a plurality of partition groups, wherein a partition group of the plurality of partition groups corresponds to a node of a first plurality of nodes and comprises a subset of data items stored in the node;
formatting the set of data items into a recovery file in accordance with the partition group index; and
loading the set of data items included in the recovery file onto a second plurality of nodes that are configured to restore the set of data items in accordance with the partition group index.
2. The method of claim 1 , further comprising:
recovering the plurality of nodes by:
identifying the set of data items stored in the node based on mappings between the plurality of partition groups and the first plurality of nodes; and
loading the set of data items onto the node.
3. The method of claim 1 , further comprising:
generating per node data for one or more nodes of the second plurality of nodes based at least in part on mappings between the plurality of partition groups and the first plurality of nodes.
4. The method of claim 1 , wherein the second plurality of nodes include fewer nodes than the first plurality of nodes, the method further comprising:
generating a partition group-to-node mapping for the second plurality of nodes from the partition group index.
5. The method of claim 1 , further comprising:
identifying one or more duplicate data items of the set of data items;
deduplicating the one or more duplicate data items;
repackaging the one or more duplicate data items into respective deduplicated data units; and
storing the respective deduplicated data units to a secondary data repository.
6. The method of claim 5 , further comprising:
determining a degree of duplicates of the set of data items; and
comparing the degree of duplicates to a threshold level of consistency, wherein the deduplication is performed responsive to the degree of duplicates being greater than the threshold level of consistency.
7. The method of claim 5 , wherein storing the respective deduplicated data units comprises:
storing a data version of the set of data items, the method further comprising:
compiling the respective deduplicated data units into the data version of the set of data items.
8. The method of claim 1 , wherein the plurality of partition groups include deduplicated data items.
9. The method of claim 1 , wherein the set of data items are stored in a not only structured query language (NoSQL) data store.
10. The method of claim 1 , further comprising:
scanning the set of data items stored in the first plurality of nodes, the first plurality of nodes comprising a first cluster of nodes; and
creating the partition group index while scanning the set of data items.
11. The method of claim 1 , further comprising:
storing the partition group index;
identifying the set of data items included in the plurality of partition groups in accordance with the partition group index; and
instantiating a second cluster of nodes using the second plurality of nodes based at least in part on the partition group index.
12. An apparatus for sub-cluster recovery in a data storage environment having a plurality of nodes, the apparatus comprising:
a processor;
memory coupled with the processor; and
instructions stored in the memory and executable by the processor to cause the apparatus to:
create a partition group index that indexes a set of data items into a plurality of partition groups, wherein a partition group of the plurality of partition groups corresponds to a node of a first plurality of nodes and comprises a subset of data items stored in the node;
format the set of data items into a recovery file in accordance with the partition group index; and
load the set of data items included in the recovery file onto a second plurality of nodes that are configured to restore the set of data items in accordance with the partition group index.
13. The apparatus of claim 12 , wherein the instructions are further executable by the processor to cause the apparatus to:
recover the plurality of nodes by:
identifying the set of data items stored in the node based on mappings between the plurality of partition groups and the first plurality of nodes; and
loading the of data items onto the node.
14. The apparatus of claim 12 , wherein the instructions are further executable by the processor to cause the apparatus to:
generate per node data for one or more nodes of the second plurality of nodes based at least in part on mappings between the plurality of partition groups and the first plurality of nodes.
15. The apparatus of claim 12 , wherein the second plurality of nodes include fewer nodes than the first plurality of nodes, wherein the instructions are further executable by the processor to cause the apparatus to:
generate a partition group-to-node mapping for the second plurality of nodes from the partition group index.
16. The apparatus of claim 12 , wherein the instructions are further executable by the processor to cause the apparatus to:
identify one or more duplicate data items of the set of data items;
deduplicate the one or more duplicate data items;
repackage the one or more duplicate data items into respective deduplicated data units; and
store the respective deduplicated data units to a secondary data repository.
17. The apparatus of claim 16 , wherein the instructions are further executable by the processor to cause the apparatus to:
determine a degree of duplicates of the set of data items; and
compare the degree of duplicates to a threshold level of consistency, wherein the deduplication is performed responsive to the degree of duplicates being greater than the threshold level of consistency.
18. The apparatus of claim 16 , wherein the instructions are further executable by the processor to cause the apparatus to:
store a data version of the set of data items; and
compile the respective deduplicated data units into the data version of the set of data items.
19. The apparatus of claim 12 , wherein the plurality of partition groups include deduplicated data items.
20. A non-transitory computer-readable medium storing code for sub-cluster recovery in a data storage environment having a plurality of nodes, the code comprising instructions executable by a processor to:
create a partition group index that indexes a set of data items into a plurality of partition groups, wherein a partition group of the plurality of partition groups corresponds to a node of a first plurality of nodes and comprises a subset of data items stored in the node;
format the set of data items into a recovery file in accordance with the partition group index; and
load the set of data items included in the recovery file onto a second plurality of nodes that are configured to restore the set of data items in accordance with the partition group index.