Forming lightweight snapshots for lossless data restore operations
Systems for high performance restore of data to storage devices. A method embodiment commences upon identifying a plurality of virtual disks to be grouped together into one or more consistency sets. Storage I/O commands for the plurality of virtual disks of the consistency sets are captured into multiple levels of backup data. On a time schedule, multiple levels of backup data for the virtual disks are cascaded by processing data from one or more higher granularity levels of backup data to one or more lower granularity levels of backup data. A restore operation can access the multiple levels of backup data to construct a restore set that is consistent to a designated point in time or to a designated state. Multiple staging areas can be maintained using lightweight snapshot data structures that each comprise a series of captured I/Os to be replayed over other datasets to generate a restore set.
1. A method comprising:
maintaining snapshot data structures at first and second granularity levels and checkpoint records at the first and second granularity levels to facilitate recovery of respective data states to a target storage area, wherein the first granularity level corresponds to a first time period and the second granularity level corresponds to a second time period, and the second time period is longer than the first time period;
wherein respective snapshot data structures at the first and second granularity levels are generated by:
storing, in a respective snapshot data structure, a plurality of storage input/output (I/O) commands received during a corresponding time period,
wherein storage I/O commands comprise commands to write to a storage area of a computing system, and
wherein respective checkpoint records at the first and second granularity levels are generated by:
replaying storage I/O commands from a particular snapshot data structure over a previously generated checkpoint record to generate a respective checkpoint record and the respective checkpoint record comprises a backup of a respective data state of a respective storage area of a respective computing system at a respective time.
2. The method of claim 1 , wherein deleting a previously generated checkpoint record, and a snapshot data structure used to generate the previously generated checkpoint record after generating a later checkpoint record.
3. The method of claim 1 , further comprising initializing a new snapshot data structure to hold at least one of, a pointer to a buffer that holds storage I/O commands, or a pointer to a storage I/O command stream.
4. The method of claim 1 , further comprising analyzing a characteristic of an incoming one of the plurality of storage I/O commands to determine a type of storage of a corresponding snapshot entry.
5. The method of claim 4 , wherein the type of storage is at least one of, a pointer to a memory location in a buffer, an identifier to a log entry in a storage I/O stream, or a copy of the incoming one of the plurality of storage I/O commands.
6. The method of claim 1 , further comprising deleting a lowest granularity checkpoint after it is applied to a full snapshot, wherein the storage area of the computing system is a storage pool that stores the full snapshot, and the lowest granularity checkpoint is applied to the full snapshot, the full snapshot maintaining a data state at a given time.
7. The method of claim 1 , wherein the plurality of storage I/O commands is produced by a virtual machine that is a member of a consistency group, wherein the snapshot data structures, and the checkpoint records, are generated for the consistency group.
8. A non-transitory computer readable medium having stored thereon a sequence of instructions which, when stored in memory and executed by a processor causes a set of acts comprising:
maintaining snapshot data structures at first and second granularity levels and checkpoint records at the first and second granularity levels to facilitate recovery of respective data states to a target storage area, wherein the first granularity level corresponds to a first time period and the second granularity level corresponds to a second time period, and the second time period is longer than the first time period;
wherein respective snapshot data structures at the first and second granularity levels are generated by:
storing, in a respective snapshot data structure, a plurality of storage input/output (I/O) commands received during a corresponding time period,
wherein storage I/O commands comprise commands to write to a storage area of a computing system, and
wherein respective checkpoint records at the first and second granularity levels are generated by:
replaying storage I/O commands from a particular snapshot data structure over a previously generated checkpoint record to generate a respective checkpoint record and the respective checkpoint record comprises a backup of a respective data state of a respective storage area of a respective computing system at a respective time.
9. The computer readable medium of claim 8 , wherein deleting a previously generated checkpoint record, and a snapshot data structure used to generate the previously generated checkpoint record after generating a later checkpoint record.
10. The computer readable medium of claim 8 , wherein the set of acts further comprise initializing a new snapshot data structure to hold at least one of, a pointer to a buffer that holds storage I/O commands, or a pointer to a storage I/O command stream.
11. The computer readable medium of claim 8 , wherein the set of acts further comprise analyzing a characteristic of an incoming one of the plurality of storage I/O commands to determine a type of storage of a corresponding snapshot entry.
12. The computer readable medium of claim 11 , wherein the type of storage is at least one of, a pointer to a memory location in a buffer, an identifier to a log entry in a storage I/O stream, or a copy of the incoming one of the plurality of storage I/O commands.
13. The computer readable medium of claim 11 , wherein the type of storage is a memory location in a buffer for a consistency group.
14. The computer readable medium of claim 8 , further comprising deleting a lowest granularity checkpoint after it is applied to a full snapshot, wherein the storage area of the computing system is a storage pool that stores the full snapshot, and a lowest granularity checkpoint is applied to the full snapshot, the full snapshot maintaining a data state at a given time.
15. The computer readable medium of claim 8 , wherein the plurality of storage I/O commands is produced by a virtual machine that is a member of a consistency group, wherein the snapshot data structures, and the checkpoint records are generated for the consistency group.
16. A system comprising: a storage medium having stored thereon a set of instructions; and a processor that executes the set of instructions to cause a set of acts comprising,
maintaining snapshot data structures at first and second granularity levels and checkpoint records at the first and second granularity levels to facilitate recovery of respective data states to a target storage area, wherein the first granularity level corresponds to a first time period and the second granularity level corresponds to a second time period, and the second time period is longer than the first time period;
wherein respective snapshot data structures at the first and second granularity levels are generated by:
storing, in a respective snapshot data structure, a plurality of storage input/output (I/O) commands received during a corresponding time period, wherein storage I/O commands comprise commands to write to a storage area of a computing system, and
wherein respective checkpoint records at the first and second granularity levels are generated by:
replaying storage I/O commands from a particular snapshot data structure over a previously generated checkpoint record to generate a respective checkpoint record and the respective checkpoint record comprises a backup of a respective data state of a respective storage area of a respective computing system at a respective time.
17. The system of claim 16 , wherein deleting a previously generated checkpoint record, and a snapshot data structure used to generate the previously generated checkpoint record after generating a later checkpoint record.
18. The system of claim 16 , wherein the set of acts further comprise initializing a new snapshot data structure to hold at least one of, a pointer to a buffer that holds storage I/O commands, or a pointer to a storage I/O command stream.
19. The system of claim 16 , wherein the set of acts further comprise analyzing a characteristic of an incoming one of the plurality of storage I/O commands to determine a type of storage of a corresponding snapshot entry.
20. The system of claim 19 , wherein the type of storage is at least one of, a pointer to a memory location in a buffer, an identifier to a log entry in a storage I/O stream, or a copy of the incoming one of the plurality of storage I/O commands.
21. The system of claim 16 , further comprising deleting a lowest granularity checkpoint after it is applied to a full snapshot, wherein the storage area of the computing system is a storage pool that stores a full snapshot, and a lowest granularity checkpoint is applied to the full snapshot, the full snapshot maintaining a data state at a given time.
22. The system of claim 16 , wherein the plurality of storage I/O commands is produced by a virtual machine that is a member of a consistency group, wherein the snapshot data structures, and the checkpoint records, are generated for the consistency group.