Stream level uninterrupted restore operation using data probe
Methods and systems for restoring data from a target device are described. According to some embodiments, the method receives a first set of data packets for restore, where the first set of data packets includes a multiplicity of data chunks. The method further captures footprints of the first set of data packets in a cache disk array. In response to receiving an acknowledgement from the cache disk array indicating the footprints of the first set of data packets have been captured, the method pushes each data chunk of the first set of data packets to a construction container for reconstruction of backup data. In response to receiving an acknowledgement from the construction container indicating the data chunk is successfully pushed, the method flushes the respective footprint of the data chunk from the cache disk array.
1. A computer-implemented method for restoring data from a target device onto a source device by a restore agent on a storage system, comprising:
using cache availability on the source device to emulate a cache disk array to be utilized by the restore agent;
receiving, by a read latch, a first set of data packets from the target device for restore, wherein the first set of data packets comprises a plurality of data chunks;
capturing footprints of the first set of data packets in the cache disk array; and
in response to receiving an acknowledgement from the cache disk array indicating that the footprints of the first set of data packets have been captured:
pushing each data chunk of the first set of data packets to a construction container on the storage system for reconstruction of backup data;
in response to receiving an acknowledgement from the construction container indicating the data chunk is successfully pushed, flushing a footprint of the data chunk from the cache disk array; and
in response to receiving an unexpected abort signal from the read latch indicating a read operation is aborted, sending a freeze signal from the restore agent to the cache disk array thereby freezing the cache disk array at a point in time where a last data chunk was successfully pushed to the construction container.
2. The method of claim 1 , further comprising: in response to receiving a trigger for a next restore,
querying remaining footprints of the first set of data packets from the cache disk array, and
identifying the last data chunk that was successfully pushed to the construction container based on the queried footprints.
3. The method of claim 2 , further comprising: in response to receiving a trigger for a next restore,
performing a health check of container data in the construction container,
if recent data chunks from the container data are insufficient to be reconstructed, re-querying the recent data chunks from the target device as a starting point for the next restore,
otherwise if the recent data chunks are sufficient to reconstruct, using the identified last data chunk as the starting point.
4. The method of claim 3 , further comprising: unfreezing the cache disk array.
5. The method of claim 1 , further comprising: when the restore is completed, receiving a second set of data packets for restore, and capturing footprints of the second set of data packets in the cache disk array.
6. The method of claim 1 , wherein the footprints of the first set of data packets include last byte information, logical address on a storage, sector information, and packet metadata information.
7. The method of claim 1 , further comprising:
indicating the read latch to hold a second set of data packets at the read latch.
8. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for, the operations comprising:
using a restore agent on a storage system to determine cache availability on a source device and emulating a cache disk array on the source device to be utilized by the restore agent;
receiving, by a read latch, a first set of data packets from a target device for restore, wherein the first set of data packets comprises a plurality of data chunks;
capturing footprints of the first set of data packets in the cache disk array; and
in response to receiving an acknowledgement from the cache disk array indicating that the footprints of the first set of data packets have been captured:
pushing each data chunk of the first set of data packets to a construction container on the storage system for reconstruction of backup data;
in response to receiving an acknowledgement from the construction container indicating the data chunk is successfully pushed, flushing a footprint of the data chunk from the cache disk array; and
in response to receiving an unexpected abort signal from the read latch indicating a read operation is aborted, sending a freeze signal from the restore agent to the cache disk array thereby freezing the cache disk array at a point in time where a last data chunk was successfully pushed to the construction container.
9. The non-transitory machine-readable medium of claim 8 , wherein the operations further comprise: in response to receiving a trigger for a next restore,
querying remaining footprints of the first set of data packets from the cache disk array, and
identifying the last data chunk that was successfully pushed to the construction container based on the queried footprints.
10. The non-transitory machine-readable medium of claim 9 , wherein the operations further comprise: in response to receiving a trigger for a next restore,
performing a health check of container data in the construction container,
if recent data chunks from the container data are insufficient to be reconstructed, re-querying the recent data chunks from the target device as a starting point for the next restore,
otherwise if the recent data chunks are sufficient to reconstruct, using the identified last data chunk as the starting point.
11. The non-transitory machine-readable medium of claim 10 , wherein the operations further comprise: unfreezing the cache disk array.
12. The non-transitory machine-readable medium of claim 8 , wherein the operations further comprise: when the restore is completed, receiving a second set of data packets for restore, and capturing footprints of the second set of data packets in the cache disk array.
13. The non-transitory machine-readable medium of claim 8 , wherein the footprints of the first set of data packets include last byte information, logical address on a storage, sector information, and packet metadata information.
14. The non-transitory machine-readable medium of claim 8 , wherein the operations further comprise: indicating the read latch to hold a second set of data packets at the read latch.
15. A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations comprising:
executing a restore agent on a storage system;
using the restore agent to determine cache availability on a source device and emulating a cache disk array on the source device to be utilized by the restore agent;
receiving, by a read latch, a first set of data packets from a target device for restore, wherein the first set of data packets comprises a plurality of data chunks;
capturing footprints of the first set of data packets in the cache disk array; and
in response to receiving an acknowledgement from the cache disk array indicating that the footprints of the first set of data packets have been captured:
pushing each data chunk of the first set of data packets to a construction container on the storage system for reconstruction of backup data;
in response to receiving an acknowledgement from the construction container indicating the data chunk is successfully pushed, flushing a footprint of the data chunk from the cache disk array; and
in response to receiving an unexpected abort signal from the read latch indicating a read operation is aborted, sending a freeze signal from the restore agent to the cache disk array thereby freezing the cache disk array at a point in time where a last data chunk was successfully pushed to the construction container.
16. The data processing system of claim 15 , wherein the operations further comprise: in response to receiving a trigger for a next restore,
querying remaining footprints of the first set of data packets from the cache disk array, and
identifying the last data chunk that was successfully pushed to the construction container based on the queried footprints.
17. The data processing system of claim 16 , wherein the operations further comprise: in response to receiving a trigger for a next restore,
performing a health check of container data in the construction container,
if recent data chunks from the container data are insufficient to be reconstructed, re-querying the recent data chunks from the target device as a starting point for the next restore,
otherwise if the recent data chunks are sufficient to reconstruct, using the identified last data chunk as the starting point.
18. The data processing system of claim 17 , wherein the operations further comprise: unfreezing the cache disk array.
19. The data processing system of claim 15 , wherein the operations further comprise: when the restore is completed, receiving a second set of data packets for restore, and capturing footprints of the second set of data packets in the cache disk array.
20. The data processing system of claim 15 , wherein the footprints of the first set of data packets include last byte information, logical address on a storage, sector information, and packet metadata information.
21. The data processing system of claim 15 , wherein the operations further comprise: indicating the read latch to hold a second set of data packets at the read latch.