Using a storage log to generate an incremental backup
An identification of a new primary snapshot created for a primary storage system is received. A threshold time window based on a capture time associated with the new primary snapshot is determined. Entries of a storage log of the primary storage system occurring within the threshold time window are analyzed to identify any objects of the primary storage system that have changed during the threshold time window. Changed objects to capture in a new secondary backup stored at a secondary storage system and corresponding to the new primary snapshot are identified. Metadata of the new primary snapshot and metadata of a previous secondary backup are compared to determine for each of the objects of the primary storage system identified as having changed during the threshold time window whether a change to an object since the previous secondary backup is captured in the new primary snapshot.
1. A method, comprising:
receiving an identification of a new primary snapshot created for a primary storage system;
determining a threshold time window based on a capture time associated with the new primary snapshot, wherein:
the capture time is generated using a snapshot service clock;
the threshold time window includes a pre-new primary snapshot marker and a post-new primary snapshot marker;
the pre-new primary snapshot marker corresponds to a first event in a storage log before the capture time;
the pre-new primary snapshot marker is determined based on a storage log clock, wherein the snapshot service clock is different from the storage log clock;
the post-new primary snapshot marker is determined based on the storage log clock; and
the post-new primary snapshot marker corresponds to a second event in the storage log after the capture time;
analyzing entries of the storage log of the primary storage system occurring within the threshold time window to identify any objects of the primary storage system that have changed during the threshold time window; and
identifying changed objects to capture in a new secondary backup stored at a secondary storage system and corresponding to the new primary snapshot, including by comparing metadata of the new primary snapshot and metadata of a previous secondary backup to determine for each of the objects of the primary storage system identified as having changed during the threshold time window whether a change to an object since the previous secondary backup is captured in the new primary snapshot, wherein the identifying of the changed objects comprises:
in response to determining that a timestamp of the snapshot service clock differs from a corresponding timestamp of the storage log clock, traversing a corresponding directory of the new primary snapshot that is associated with a changed object of the changed objects.
2. The method of claim 1 , further comprising generating an incremental backup of the primary storage system based on the identified changed objects, wherein the incremental backup of the primary storage system is the new secondary backup of the new primary snapshot.
3. The method of claim 1 , wherein the changed objects include at least one of a created file, a deleted file, a modified file, a moved file, a created directory, a deleted directory, and/or a moved directory.
4. The method of claim 1 , wherein the capture time associated with the new primary snapshot is not reflected in the storage log.
5. The method of claim 1 , wherein identifying the changed objects to capture in the new secondary backup includes adding to a change list the changed objects that are determined to be captured in the new primary snapshot.
6. The method of claim 1 , wherein the primary storage system is a network-attached storage device.
7. The method of claim 1 , further comprising determining a second threshold time window based on a capture time associated with a previous primary snapshot.
8. The method of claim 7 , wherein timestamps associated with the pre-new primary snapshot marker and the post-new primary snapshot marker are based on the storage log clock.
9. The method of claim 8 , wherein the second threshold time window includes a pre-previous primary snapshot marker and a post-previous primary snapshot marker, wherein timestamps associated with the pre-previous primary snapshot marker and the post-previous primary snapshot marker are based on the storage log clock.
10. The method of claim 9 , further comprising adding to a change list objects identified as occurring within the threshold time window or the second threshold time window that are verified as being associated with the new primary snapshot.
11. The method of claim 10 , further comprising adding to the change list objects of the primary storage system that have changed between the threshold time window and the second threshold time window.
12. The method of claim 1 , further comprising determining whether a number of the identified changed objects of the primary storage system that have changed during the threshold time window is greater than a threshold number of objects.
13. The method of claim 12 , further comprising performing a full backup of the new primary snapshot in response to determining that the number of the identified changed objects of the primary storage system that have changed during the threshold time window is greater than the threshold number of objects.
14. The method of claim 12 , wherein in response to determining that the number of the identified changed objects of the primary storage system that have changed during the threshold time window is not greater than the threshold number of objects, identifying the changed objects to capture in the new secondary backup stored at the secondary storage system and corresponding to the new primary snapshot.
15. A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
receiving an identification of a new primary snapshot created for a primary storage system;
determining a threshold time window based on a capture time associated with the new primary snapshot, wherein:
the capture time is generated using a snapshot service clock;
the threshold time window includes a pre-new primary snapshot marker and a post-new primary snapshot marker;
the pre-new primary snapshot marker corresponds to a first event in a storage log before the capture time;
the pre-new primary snapshot marker is determined based on a storage log; clock, wherein the snapshot service clock is different from the storage log clock;
the post-new primary snapshot marker is determined based on the storage log clock; and
the post-new primary snapshot marker corresponds to a second event in the storage log after the capture time;
analyzing entries of the storage log of the primary storage system occurring within the threshold time window to identify any objects of the primary storage system that have changed during the threshold time window; and
identifying changed objects to capture in a new secondary backup stored at a secondary storage system and corresponding to the new primary snapshot, including by comparing metadata of the new primary snapshot and metadata of a previous secondary backup to determine for each of the objects of the primary storage system identified as having changed during the threshold time window whether a change to an object since the previous secondary backup is captured in the new primary snapshot, wherein the identifying of the changed objects comprises:
in response to determining that a timestamp of the snapshot service clock differs from a corresponding timestamp of the storage log clock, traversing a corresponding directory of the new primary snapshot that is associated with a changed object of the changed objects.
16. A system, comprising:
a processor configured to:
receive an identification of a new primary snapshot created for a primary storage system;
determine a threshold time window based on a capture time associated with the new primary snapshot, wherein:
the capture time is generated using a snapshot service clock;
the threshold time window includes a pre-new primary snapshot marker and a post-new primary snapshot marker;
the pre-new primary snapshot marker corresponds to a first event in a storage log before the capture time;
the pre-new primary snapshot marker is determined based on a storage log clock, wherein the snapshot service clock is different from the storage log clock;
the post-new primary snapshot marker is determined based on the storage log clock; and
the post-new primary snapshot marker corresponds to a second event in the storage log after the capture time;
analyze entries of the storage log of the primary storage system occurring within the threshold time window to identify any objects of the primary storage system that have changed during the threshold time window; and
identify changed objects to capture in a new secondary backup stored at a secondary storage system and corresponding to the new primary snapshot, including by comparing metadata of the new primary snapshot and metadata of a previous secondary backup to determine for each of the objects of the primary storage system identified as having changed during the threshold time window whether a change to an object since the previous secondary backup is captured in the new primary snapshot, wherein the identifying of the changed objects comprises to:
in response to determining that a timestamp of the snapshot service clock differs from a corresponding timestamp of the storage log clock, traverse a corresponding directory of the new primary snapshot that is associated with a changed object of the changed objects; and
a memory coupled to the processor and configured to provide the processor with instructions.