Sub-cluster recovery using a partition group index

1. A method for sub-cluster recovery in a data storage environment having a plurality of nodes, the method comprising:

scanning data items stored in the first plurality of nodes of a first cluster;

while scanning, creating a partition group index, the partition group index indexing the data items into a plurality of partition groups, each partition group corresponding to a node of the first plurality of nodes and comprising a subset of data items stored in the node;

storing the index; and

instantiating a second cluster using a second plurality of nodes different, in number of nodes, from the first plurality of nodes, each node of the second plurality nodes being instantiated with data from at least one of the partition groups, the instantiating of the second cluster comprising for each node of the second plurality of nodes, generating per node data based on the mappings between the partition groups and the first plurality of nodes.

2. The method of claim 1 , further comprising:

recovering each of the first plurality of nodes by:

identifying the data items stored in the node based on the mappings between the partition groups and the first plurality of nodes, and

restoring the identified data items onto the node.

3. The method of claim 1 , wherein the second plurality of nodes includes less nodes than the first plurality of nodes and wherein the instantiating the second cluster comprises:

generating a partition group to node mapping for the second cluster from the partition group index.

4. The method of claim 1 , further comprising:

while scanning the data items, identifying duplicate data items in the cluster;

deduplicating the duplicate data items;

repackaging each of the duplicate data items into respective deduplicated data units; and

storing the deduplicated data units to a secondary data repository.

5. The method of claim 4 , further comprising:

determining a degree of duplicates of the data items in the cluster; and

comparing the degree of duplicates to a predetermined level of consistency,

wherein the deduplication is performed responsive to determining the degree of duplicates is greater than the predetermined level of consistency.

6. The method of claim 4 , wherein the partition groups include deduplicated data items.

7. The method of claim 4 , wherein storing the deduplicated data units comprises:

storing a data version of the data items and;

compiling the deduplicated data units into the data version of the data items.

8. The method of claim 1 , wherein the data items are stored in a No SQL data store.

9. The method of claim 1 , wherein a partition group is a group of data at is replicated across multiple nodes.

10. A system for sub-cluster recovery in a data storage environment having a plurality of nodes, the system comprising:

one or more non-transitory computer readable storage media;

a processing system operatively coupled with the one or more computer readable storage media; and

program instructions stored on the one or more computer readable storage media that, when read and executed by the processing system; direct the processing system to perform operations comprising:

scanning data items stored in the first plurality of nodes of a first cluster;

while scanning, creating a partition group index, the partition group index indexing the data items into a plurality of partition groups, each partition group corresponding to a node of the first plurality of nodes and comprising a subset of data items stored in the node;

storing the index; and

instantiating a second cluster using a second plurality of nodes different; in number of nodes, from the first plurality of nodes, each node of the second plurality nodes being instantiated with data from at least one of the partition groups, the instantiating of the second cluster comprising for each node of the second plurality of nodes, generating per node data based on the mappings between the partition groups and the first plurality of nodes.

11. The system of claim 10 , wherein the program instructions further direct the processing system to perform operations comprising:

recovering each of the plurality of nodes by:

identifying the data items stored in the node based on the mappings between the partition groups and the first plurality of nodes, and

restoring the identified data items onto the node.

12. The system of claim 10 , wherein the second plurality of nodes includes less nodes than the first plurality of nodes and wherein instantiating the second cluster includes generating a partition group to node mapping for the second cluster from the partition group index.

13. The system of claim 10 , wherein the program instructions further direct the processing system to perform operations comprising:

while scanning the data items, identify duplicate data items in the cluster;

duplicating the duplicate data items;

repackaging each of the duplicate data items into respective deduplicated data units; and

storing the deduplicated data units to a secondary data repository.

14. The system of claim 13 , wherein the program instructions further direct the processing system to perform operations comprising:

determining a degree of duplicates of the data items in the cluster; and

comparing the degree of duplicates to a predetermined level of consistency,

wherein the deduplicating is performed responsive to determining the degree of duplicates is greater than the predetermined level of consistency.

15. The system of claim 13 , wherein the partition groups include deduplicated data items.

16. The system of claim 13 , wherein the storing the deduplicated data units comprises storing a data version of the data items, and wherein the program instructions further direct the processing system to perform operations comprising:

compiling the deduplicated data units into the data version of the data items.

17. The system of claim 10 , wherein the data items are stored in a No SQL data store.

18. The system of claim 10 , wherein a partition group is a group of data that is replicated across multiple nodes.

19. A method for sub-cluster recovery in a data storage environment having a plurality of nodes, the method comprising:

scanning data items stored in the first plurality of nodes of a first cluster;

while scanning, creating a partition group index, the partition group index indexing the data items into a plurality of partition groups, each partition group corresponding to a node of the first plurality of nodes and comprising a subset of data items stored in the node;

storing the index; and

instantiating a second cluster using a second plurality of nodes different, in number of nodes, from the first plurality of nodes, each node of the second plurality nodes being instantiated with data from at least one of the partition groups, the second plurality of nodes including less nodes than the first plurality of nodes, the instantiating of the second cluster comprising generating a partition group to node mapping for the second cluster from the partition group index.

20. A system for sub-cluster recover in a data storage environment having a plurality of nodes, the system comprising:

one or more non-transitory computer readable storage media;

a processing system operatively coupled with the one or more computer readable storage media; and

program instructions stored on the one or more computer readable storage media that, when read and executed by the processing system, direct the processing system to perform operations comprising:

scanning data items stored in the first plurality of nodes of a first cluster;

while scanning, creating a partition group index, the partition group index indexing the data items into a plurality of partition groups, each partition group corresponding to a node of the first plurality of nodes and comprising a subset of data items stored in the node;

storing the index; and

instantiating a second cluster using a second plurality of nodes different, in number of nodes, from the first plurality of nodes, each node of the second plurality nodes being instantiated with data from at least one of the partition groups, the second plurality of nodes including less nodes than the first plurality of nodes, the instantiating of the second cluster comprising generating a partition group to node mapping for the second cluster from the partition group index.

21. A method for sub-cluster recovery in a data storage environment having a plurality of nodes, the method comprising:

scanning data items stored in the first plurality of nodes of a first cluster;

while scanning, creating a partition group index and simultaneously identifying duplicate data items in the cluster;

indexing the data items into a plurality of partition groups, each partition group corresponding to a node of the first plurality of nodes and comprising a subset of data items stored in the node;

storing the index;

deduplicating the duplicate data items;

repackaging each of the duplicate data items into respective deduplicated data units, wherein a deduplicated data unit comprises a data form that includes a single instance of a deduplicated data item and information corresponding to a location from which the deduplicated data item originated;

storing the deduplicated data units to a secondary data repository; and

instantiating a second cluster using a second plurality of nodes different, in number of nodes, from the first plurality of nodes, each node of the second plurality nodes being instantiated with data from at least one of the partition groups, the second plurality of nodes including less nodes than the first plurality of nodes, the instantiating of the second cluster comprising generating a partition group to node mapping for the second cluster from the partition group index.