Methods, systems, and apparatuses to update point in time journal using map reduce to create a highly parallel update

1. A storage system, comprising:

a processor;

a storage device in operable communication with the processor;

a replication site in operable communication with the processor, the replication site configured to be in operable communication with a production site; and

an object store configured for the replication site, the object store comprising a plurality of data objects associated with data stored in at least one logical unit (LU) of the production site, a plurality of metadata objects, and a plurality of change objects;

wherein the replication site is configured to generate a requested point in time (PIT) based at least in part on the plurality of data objects, the generation of the PIT comprising:

dividing a received stream comprising the plurality of metadata objects, ordered by time, into a plurality of respective portions of metadata objects, wherein each respective metadata object in the stream further comprises an I/O block offset in the LU, and wherein each of the plurality of respective portions of metadata objects are provided to a respective mapping node;

remapping, at each respective mapper node, the received portion of metadata objects, such that the metadata objects in the received portion are ordered by their respective I/O block offsets;

providing, based on the remapping, each respective metadata object, based on its respective I/O block offset within the LU, to a respective one of a plurality of reducer nodes in operable communication with the storage system, wherein the reducer node is assigned at least to the respective I/O block offset;

performing a reduce operation on the received metadata objects, by each respective one of the plurality of reducer nodes, to retain, at the respective I/O block offset, only the most recent change to the metadata before the requested PIT, wherein the reduce operation further comprises reading data from the plurality of change objects based at least in part on the locations in the one or more LUs where change has occurred, wherein the data is read substantially concurrently and in parallel; and

merging together the list of changes from each reducer node into the requested PIT;

wherein the generation of the PIT further comprises performing at least one of the map and reduce operations substantially concurrently and in parallel, at each respective one of the respective map or reducer nodes.

2. The storage system of claim 1 , wherein the generation of the requested PIT further comprises, substantially concurrently and in parallel:

receiving at each respective reducer node, a list of the most recent changes that occurred at each I/O block offset before the desired PIT; and

performing the reduce operation at each respective reducer node, to retain only the most recent change to the metadata before the requested PIT.

3. The storage system of claim 1 , wherein the generation of the PIT further comprises, substantially concurrently and in parallel:

applying changes from the change object to the plurality of data objects; and

saving a new copy of the plurality of data objects as the PIT.

4. The storage system of claim 1 , wherein at least a portion of at least one of the reducer and mapper nodes comprise virtual machines that are generated in response to receiving a request for a PIT.

5. The storage system of claim 1 , wherein at least a portion of the reducer nodes are the same as at least a portion of the mapper nodes.

6. The storage system of claim 1 , wherein at least a portion of the reducer nodes comprises nodes that are different than the first plurality of mapper nodes.

7. The storage system of claim 1 , wherein, at each respective reducer node, the reduce operation comprises;

ordering the metadata objects, associated with the respective offset, by time;

retaining at the respective offset, based on the ordering by time, only the latest metadata objects prior to the requested PIT, so that for each respective offset, there is only one metadata entry;

reading, from one or more of the plurality of change objects, associated data that the latest metadata object describes; and

applying, to corresponding disk data objects, changes from the corresponding data change object relating to the most recent change that was retained.

8. A computer-implemented method, comprising:

providing a process in operable communication with a storage device;

generating an object store disposed at a replication site in operable communication with the processor, the object store comprising a plurality of data objects associated with data stored in at least one logical unit (LU) of a production site in operable communication with the replication site, a plurality of metadata objects, and a plurality of change objects;

configuring the replication site to generate a requested point in time (PIT) based at least in part on the plurality of data objects, the generation of the PIT comprising:

dividing a received stream comprising the plurality of metadata data objects, ordered by time, into a plurality of respective portions of metadata objects, wherein each respective metadata object in the stream further comprises an I/O block offset in the LU, and wherein each of the plurality of respective portions of metadata objects are provided to a respective mapping node;

remapping, at each respective mapper node, the received portion of metadata objects, such that the metadata objects in the received portion are ordered by their respective I/O block offsets;

providing, based on the remapping, each respective metadata object, based on its respective I/O block offset within the LU, to a respective one of a plurality of reducer nodes in operable communication with the storage system, wherein the reducer node is assigned at least to the respective I/O block offset;

performing a reduce operation on the received metadata objects, by each respective one of the plurality of reducer nodes, to retain, at the respective I/O block offset, only the most recent change to the metadata before the requested PIT, wherein the reduce operation further comprises reading data from the plurality of change objects based at least in part on the locations in the one or more LUs where change has occurred, wherein the data is read substantially concurrently and in parallel; and

merging together the list of changes from each respective reducer node into the requested PIT

wherein the generation of the PIT further comprises performing at least one of the map and reduce operations substantially concurrently and in parallel, at each respective one of the respective map or reducer nodes.

9. The computer-implemented method of claim 8 , further comprising, substantially concurrently and in parallel:

receiving at each respective reducer node, a list of the most recent changes that occurred at each I/O block offset before the desired PIT; and

performing the reduce operation at each respective reducer node to retain only the most recent change to the metadata before the requested PIT.

10. The computer-implemented method of claim 8 , further comprising, substantially concurrently and in parallel:

applying changes from the change object to the plurality of data objects; and

saving a new copy of the plurality of data objects as the PIT.

11. The computer-implemented method of claim 8 , wherein at least a portion of at least one of the reducer and mapper nodes comprise virtual machines that are generated in response to receiving a request for a PIT.

12. The computer-implemented method of claim 8 , wherein at least a portion of the reducer nodes are the same as at least a portion of the mapper nodes.

13. The computer-implemented method of claim 8 , wherein, at each respective reducer node, the reduce operation comprises;

ordering the metadata objects, associated with the respective offset, by time;

retaining at the respective offset, based on the ordering by time, only the latest metadata objects prior to the requested PIT, so that for each respective offset, there is only one metadata entry;

reading, from one or more of the plurality of change objects, associated data that the latest metadata object describes; and

applying, to corresponding disk data objects, changes from the corresponding data change object relating to the most recent change that was retained.

14. A computer program product including a non-transitory computer readable storage medium having computer program code encoded thereon that when executed on a processor of a computer causes the computer to operate a storage system, the computer program product comprising:

computer program code for providing a processor;

computer program code for providing a storage device in operable communication with the processor;

computer program code for generating an object store disposed at a replication site in operable communication with the processor, the object store comprising a plurality of data objects associated with data stored in at least one logical unit (LU) of a production site in operable communication with the replication site, a plurality of metadata objects, and a plurality of change objects;

computer program code for configuring the replication site to generate a requested point in time (PIT) based at least in part on the plurality of data objects;

computer program code for dividing a received stream comprising the plurality of metadata data objects, ordered by time, into a plurality of respective portions of data objects, wherein each respective metadata object in the stream further comprises an I/O block offset in the LU, and wherein each of the plurality of respective portions of metadata objects are provided to a respective mapping node;

computer program code for remapping, at each respective mapper node, the received portion of metadata objects, such that the metadata objects in the received portion are ordered by their respective I/O block offsets;

computer program code for providing, based on the remapping, each respective metadata object, based on its respective I/O block offset within the LU, to a respective one of a plurality of reducer nodes in operable communication with the storage system, wherein the reducer node is assigned at least to the respective I/O block offset;

computer program code for performing a reduce operation on the received metadata objects, by each respective one of the plurality of reducer nodes, to retain, at the respective I/O block offset, only the most recent change to the metadata before the requested PIT, wherein the reduce operation further comprises reading data from the plurality of change objects based at least in part on the locations in the one or more LUs where change has occurred, wherein the data is read substantially concurrently and in parallel;

computer program code for merging together the list of changes from each respective reducer node into the requested PIT; and

computer program code for performing at least one of the map and reduce operations substantially concurrently and in parallel, at each respective one of the respective map or reducer nodes.

15. The computer program product of claim 14 , further comprising:

computer program code for receiving at each respective reducer node, substantially concurrently and in parallel, a list of the most recent changes that occurred at each offset before the desired PIT;

computer program code for performing a reduce operation at each respective reducer node, substantially concurrently and in parallel, to retain only the most recent change to the metadata before the requested PIT;

computer program code for applying changes from the change object to the plurality of data objects, substantially concurrently and in parallel; and

computer program code for saving, substantially concurrently and on parallel, a new copy of the plurality of data objects as the PIT.

16. The computer program product of claim 14 , wherein, at each respective reducer node, the reduce operation comprises;

computer program code for ordering the metadata objects, associated with the respective offset, by time;

computer program code for retaining at the respective offset, based on the ordering by time, only the latest metadata objects prior to the requested PIT, so that for each respective offset, there is only one metadata entry;

computer program code for reading, from one or more of the plurality of change objects, associated data that the latest metadata object describes; and

computer program code for applying, to corresponding disk data objects, changes from the corresponding data change object relating to the most recent change that was retained.