Sub-cluster recovery using a partition group index
Embodiments disclosed herein provide systems, methods, and computer readable media for sub-cluster recovery in a data storage environment having a plurality of storage nodes. In a particular embodiment, the method provides scanning data items in the plurality of nodes. While scanning, the method further provides indexing the data items into an index of a plurality of partition groups. Each partition group includes data items owned by a particular one of the plurality of storage nodes. The method then provides storing the index.
1. A method comprising:
creating a partition group index that performs an indexing of a set of data items into a plurality of partition groups;
deduplicating, concurrently with the indexing, the set of data items stored in a node of a plurality of nodes into a set of deduplicated data items comprising singular copies of each data item of the set of data items, wherein a partition group of the plurality of partition groups corresponds to a node of a first plurality of nodes and comprises a subset of data items of the set of deduplicated data items stored in the node;
formatting the set of deduplicated data items into a recovery file in accordance with the partition group index; and
loading the set of deduplicated data items included in the recovery file onto a second plurality of nodes that are configured to restore the set of deduplicated data items in accordance with the partition group index.
2. The method of claim 1 , further comprising:
recovering the plurality of nodes by:
identifying the set of deduplicated data items stored in the node based on mappings between the plurality of partition groups and the first plurality of nodes; and
loading the set of deduplicated data items onto the node.
3. The method of claim 1 , wherein a deduplicated data item of the set of deduplicated data items comprises a data form that includes a single instance of the deduplicated data item and an indication of one or more original file locations associated with the deduplicated data item, the method further comprising:
accessing the one or more original file locations for recovery based at least in part on the data form of the deduplicated data item.
4. The method of claim 1 , further comprising:
generating per node data for each node one or more nodes of the second plurality of nodes based at least in part on mappings between the plurality of partition groups and the first plurality of nodes.
5. The method of claim 1 , wherein the second plurality of nodes include fewer nodes than the first plurality of nodes, the method further comprising:
generating a partition group-to-node mapping for the second plurality of nodes from the partition group index.
6. The method of claim 1 , wherein deduplicating the set of data items comprises:
identifying one or more duplicate data items of the set of data items;
deduplicating the one or more duplicate data items;
repackaging the one or more duplicate data items into the set of deduplicated data items; and
storing the set of deduplicated data items to a secondary data repository.
7. The method of claim 6 , further comprising:
determining a degree of duplicates of the set of data items; and
comparing the degree of duplicates to a threshold level of consistency, wherein deduplication is performed responsive to the degree of duplicates being greater than the threshold level of consistency.
8. The method of claim 6 , wherein storing the set of deduplicated data items comprises storing a data version of the set of data items, the method further comprising:
compiling the set of deduplicated data items into the data version of the set of data items.
9. The method of claim 6 , wherein repackaging the one or more duplicate data items comprises:
storing the set of deduplicated data items into a data file for storage in the secondary data repository.
10. The method of claim 1 , wherein the plurality of partition groups include the set of deduplicated data items.
11. The method of claim 1 , wherein the set of deduplicated data items are stored in a not only structured query language (NoSQL) data store.
12. The method of claim 1 , further comprising:
scanning the set of data items stored in the first plurality of nodes, the first plurality of nodes comprising a first cluster of nodes; and
creating the partition group index while scanning the set of data items, wherein the deduplication occurs concurrently with the scanning.
13. The method of claim 1 , further comprising:
storing the partition group index;
identifying the set of deduplicated data items included in the plurality of partition groups in accordance with the partition group index; and
instantiating a second cluster of nodes using the second plurality of nodes based at least in part on the partition group index.
14. An apparatus comprising:
at least one processor;
at least one memory coupled with the at least one processor; and
instructions stored in the at least one memory and executable by the at least one processor to cause the apparatus to:
create a partition group index that performs an indexing of a set of data items into a plurality of partition groups;
deduplicate, concurrently with the indexing, the set of data items stored in a node of a plurality of nodes into a set of deduplicated data items comprising singular copies of each data item of the set of data items, wherein a partition group of the plurality of partition groups corresponds to a node of a first plurality of nodes and comprises a subset of data items of the set of deduplicated data items of the set of deduplicated data items stored in the node;
format the set of deduplicated data items into a recovery file in accordance with the partition group index; and
load the set of deduplicated data items included in the recovery file onto a second plurality of nodes that are configured to restore the set of deduplicated data items in accordance with the partition group index.
15. The apparatus of claim 14 , wherein the instructions are further executable by the at least one processor to cause the apparatus to:
recover the plurality of nodes by:
identifying the set of deduplicated data items stored in the node based on mappings between the plurality of partition groups and the first plurality of nodes; and
loading the set of deduplicated data items onto the node.
16. The apparatus of claim 15 , wherein a deduplicated data item of the set of deduplicated data items comprises a data form that includes a single instance of the deduplicated data item and an indication of one or more original file locations associated with the deduplicated data item, wherein the instructions are further executable by the at least one processor to cause the apparatus to:
access the one or more original file locations for recovery based at least in part on the data form of the deduplicated data item.
17. The apparatus of claim 14 , wherein the instructions are further executable by the at least one processor to cause the apparatus to:
generate per node data for each node one or more nodes of the second plurality of nodes based at least in part on mappings between the plurality of partition groups and the first plurality of nodes.
18. The apparatus of claim 14 , wherein the instructions are further executable by the at least one processor to cause the apparatus to:
identify one or more duplicate data items of the set of data items;
deduplicate the one or more duplicate data items;
repackage the one or more duplicate data items into the set of deduplicated data items; and
store the set of deduplicated data items to a secondary data repository.
19. The apparatus of claim 18 , wherein the instructions are further executable by the at least one processor to cause the apparatus to:
determine a degree of duplicates of the set of data items; and
compare the degree of duplicates to a threshold level of consistency, wherein deduplication is performed responsive to the degree of duplicates being greater than the threshold level of consistency.
20. A non-transitory computer-readable medium storing code, the code comprising instructions executable by at least one processor to:
create a partition group index that performs an indexing of a set of data items into a plurality of partition groups;
deduplicate, concurrently with the indexing, the set of data items stored in a node of a plurality of nodes into a set of deduplicated data items comprising singular copies of each data item of the set of data items, wherein a partition group of the plurality of partition groups corresponds to a node of a first plurality of nodes and comprises a subset of data items of the set of deduplicated data items stored in the node;
format the set of deduplicated data items into a recovery file in accordance with the partition group index; and
load the set of deduplicated data items included in the recovery file onto a second plurality of nodes that are configured to restore the set of deduplicated data items in accordance with the partition group index.