Efficient content indexing of incremental block-based backups
A first list is received including locations of blocks of a file system that changed between current and previous backups. A superblock of the file system is read to generate a second list including locations of metadata records representing file objects. The first and second lists are intersected to return a third list including locations listed in the first and second lists. The locations in the third list are changed metadata records. A metadata record as of the current backup and identified in the third list is read. A previous version of the metadata record as of the previous backup is read. The current and previous metadata record versions are compared to detect potential conditions occurring between the backups. Upon detection of a first type of condition, information about the condition is recorded. Upon detection of a second type of condition, information about the condition is not recorded.
1. A method for indexing contents of an incremental block-based backup of a volume having a file system comprising:
allowing the incremental block-based backup of the volume to complete, the incremental backup being a most recent backup and comprising storing on backup storage all blocks of the volume that have changed since a previous backup according to a changed block list generated by a change block tracking (CBT) driver that intercepts writes from the file system;
after the most recent incremental backup has completed,
obtaining the changed block list used for the most recent incremental backup, the changed block list being a first list;
reading superblocks of the file system to generate a second list identifying metadata blocks at which metadata records associated with files of the file system are stored;
intersecting the first and second lists to return a third list identifying changed metadata blocks, the changed metadata blocks thereby corresponding to metadata records that have changed between the previous and most recent backups;
reading a changed metadata record, the changed metadata record being a current version of the changed metadata record;
reading another version of the changed metadata record as of the previous backup, the other version thereby being a previous version of the changed metadata record;
comparing the current and previous versions of the changed metadata records to detect events that occurred between the previous and most recent backups;
upon detecting that an event of a first type has occurred, recording, in a journal, information about the event of the first type, wherein the journal is different from the changed block list generated by the CBT driver; and
upon detecting that an event of a second type has occurred, not recording, in the journal, information about the event of the second type, wherein the event of the second type comprises a creation of a file after the previous backup and a deletion of the file before the most recent backup, wherein the detecting that the event of a first type has occurred further comprises:
reading an entry for a file object in the previous version of the changed metadata record, wherein an in-use flag field in the previous version of the changed metadata record comprises a value indicating that the previous version of the changed metadata record was in-use as of the previous backup;
reading an entry for a file object in the current version of the changed metadata record, wherein an in-use flag field in the current version of the changed metadata record comprises a value indicating that the current version of the changed metadata record was in-use as of the most recent backup;
comparing a generation counter value stored in a generation counter field of the previous version of the changed metadata record and a generation counter value stored in a generation counter field of the current version of the changed metadata record to determine whether or not the generation counter values match;
if the generation counter values match, determining that the file object associated with the current version of the changed metadata record has been modified and recording, in the journal, information about the file object having been modified; and
if the generation counter values do not match, determining that the file object associated with the previous version of the changed metadata record has been deleted and recording, in the journal, information about the file object having been deleted.
2. The method of claim 1 wherein the detecting that the event of a first type has occurred further comprises:
reading an entry for a file object in the previous version of the changed metadata record, wherein an in-use flag field in the previous version of the changed metadata record comprises a value indicating that the previous version of the changed metadata record was not in-use as of the previous backup;
reading an entry for a file object in the current version of the changed metadata record, wherein an in-use flag field in the current version of the changed metadata record comprises a value indicating that the current version of the changed metadata record was in-use as of the most recent backup;
determining that the file object associated with the current version of the changed metadata record is new; and
recording, in the journal, information about the file object having been newly created.
3. The method of claim 1 wherein the detecting that the event of a first type has occurred further comprises:
reading an entry for a file object in the previous version of the changed metadata record, wherein an in-use flag field in the previous version of the changed metadata record comprises a value indicating that the previous version of the changed metadata record was in-use as of the previous backup;
reading an entry for a file object in the current version of the changed metadata record, wherein an in-use flag field in the previous version of the changed metadata record comprises a value indicating that the current version of the changed metadata record was not in-use as of the most recent backup;
determining that the file object associated with the previous version of the changed metadata record was deleted; and
recording, in the journal, information about the file object having been deleted.
4. The method of claim 1 wherein the detecting that the event of a second type has occurred further comprises:
reading an entry for a file object in the previous version of the changed metadata record, wherein an in-use flag field in the previous version of the changed metadata record comprises a value indicating that the previous version of the changed metadata record was not in-use as of the previous backup;
reading an entry for a file object in the current version of the changed metadata record, wherein an in-use flag field in the current version of the changed metadata record comprises a value indicating that the current version of the changed metadata record was not in-use as of the most recent backup;
determining that the file object associated with the current version of the changed metadata record was created and deleted between the previous and most recent backups of the volume; and
not recording, in the journal, information about the file object having been created and deleted between the previous and most recent backups.
5. The method of claim 1 wherein all the blocks of the volume that have changed since the previous backup are backed up during the most recent backup before the intersecting.
6. A system to detect file changes between block-based backups for indexing contents of an incremental block-based backup of a volume having a file system, the system comprising:
a processor-based system executed on a computer system and configured to execute instructions comprising:
allowing the incremental block-based backup of the volume to complete, the incremental backup being a current backup and comprising storing on backup storage all blocks of the volume that have changed since a previous backup according to a changed block list generated by a change block tracking (CBT) driver that intercepts writes from the file system;
after the incremental backup has completed,
obtaining the changed block list, the changed block list being a first list;
reading superblocks of the file system to generate a second list identifying metadata blocks at which metadata records associated with files of the file system are stored;
intersecting the first and second lists to return a third list identifying changed metadata blocks, the changed metadata blocks thereby corresponding to metadata records that have changed between the previous and current backups;
reading a changed metadata record, the changed metadata record being a current version of the changed metadata record;
reading another version of the changed metadata record as of the previous backup, the other version thereby being a previous version of the changed metadata record;
comparing the current and previous versions of the changed metadata records to detect events that occurred between the previous and current backups;
upon detecting that an event of a first type has occurred, recording, in a journal, information about the event of the first type, wherein the journal is separate from the changed block list generated by the CBT driver; and
upon detecting that an event of a second type has occurred, not recording, in the journal, information about the event of the second type, wherein the event of the second type comprises a creation of a file after the previous backup and a deletion of the file before the current backup, wherein the detecting that the event of a first type has occurred further comprises:
reading an entry for a file object in the previous version of the changed metadata record, wherein an in-use flag field in the previous version of the changed metadata record comprises a value indicating that the previous version of the changed metadata record was in-use as of the previous backup;
reading an entry for a file object in the current version of the changed metadata record, wherein an in-use flag field in the current version of the changed metadata record comprises a value indicating that the current version of the changed metadata record was in-use as of the current backup;
comparing a generation counter value stored in a generation counter field of the previous version of the changed metadata record and a generation counter value stored in a generation counter field of the current version of the changed metadata record to determine whether or not the generation counter values match;
if the generation counter values match, determining that the file object associated with the current version of the changed metadata record has been modified and recording, in the journal, information about the file object having been modified; and
if the generation counter values do not match, determining that the file object associated with the previous version of the changed metadata record has been deleted and recording, in the journal, information about the file object having been deleted.
7. The system of claim 6 wherein the detecting that the event of a first type has occurred further comprises:
reading an entry for a file object in the previous version of the changed metadata record, wherein an in-use flag in the previous version of the changed metadata record comprises a value indicating that the previous version of the changed metadata record was not in-use as of the previous backup;
reading an entry for a file object in the current version of the changed metadata record, wherein an in-use flag in the current version of the changed metadata record comprises a value indicating that the current version of the changed metadata record was in-use as of the current backup;
determining that the file object associated with the current version of the changed metadata record is new; and
recording, in the journal, information about the file object having been newly created.
8. The system of claim 6 wherein the detecting that the event of a first type has occurred further comprises:
reading an entry for a file object in the previous version of the changed metadata record, wherein an in-use flag field in the previous version of the changed metadata record comprises a value indicating that the previous version of the changed metadata record was in-use as of the previous backup;
reading an entry for a file object in the current version of the changed metadata record, wherein an in-use flag field in the current version of the changed metadata record comprises a value indicating that the current version of the changed metadata record was not in-use as of the current backup;
determining that the file object associated with the previous version of the changed metadata record was deleted; and
recording, in the journal, information about the file object having been deleted.
9. The system of claim 6 wherein the detecting that the event of a second type has occurred further comprises:
reading an entry for a file object in the previous version of the changed metadata record, wherein an in-use flag field in the previous version of the changed metadata record comprises a value indicating that the previous version of the changed metadata record was not in-use as of the previous backup;
reading an entry for a file object in the current version of the changed metadata record, wherein an in-use flag field in the current version of the changed metadata record comprises a value indicating that the current version of the changed metadata record was not in-use as of the current backup;
determining that the file object associated with the current version of the changed metadata record was created and deleted between the previous and current backups of the volume; and
not recording, in the journal, information about the file object having been created and deleted between the previous and current backups.
10. The system of claim 6 wherein all the blocks of the volume that have changed since the previous backup are backed up during the current backup before the intersecting.
11. A computer program product, comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method for indexing contents of an incremental block-based backup of a volume having a file system, the method comprising:
allowing the incremental block-based backup of the volume to complete, the incremental backup being a current backup and comprising storing on backup storage all blocks of the volume that have changed since a previous backup according to a changed block list generated by a change block tracking (CBT) driver that intercepts writes from the file system;
after the incremental backup has completed,
obtaining the changed block list, the changed block list being a first list;
reading superblocks of the file system to generate a second list identifying metadata blocks at which metadata records associated with files of the file system are stored;
intersecting the first and second lists to return a third list identifying changed metadata blocks, the changed metadata blocks thereby corresponding to metadata records that have changed between the previous and current backups;
reading a changed metadata record, the changed metadata record being a current version of the changed metadata record;
reading another version of the changed metadata record as of the previous backup, the other version thereby being a previous version of the changed metadata record;
comparing the current and previous versions of the changed metadata records to detect events that occurred between the previous and current backups;
upon detecting that an event of a first type has occurred, recording, in a journal, information about the event of the first type, wherein the journal is stored separate from the changed block list; and
upon detecting that an event of a second type has occurred, not recording, in the journal, information about the event of the second type, wherein the event of the second type comprises a creation of a file after the previous backup and a deletion of the file before the current backup, wherein the detecting that the event of a first type has occurred further comprises:
reading an entry for a file object in the previous version of the changed metadata record, wherein an in-use flag field in the previous version of the changed metadata record comprises a value indicating that the previous version of the changed metadata record was in-use as of the previous backup;
reading an entry for a file object in the current version of the changed metadata record, wherein an in-use flag field in the current version of the changed metadata record comprises a value indicating that the current version of the changed metadata record was in-use as of the current backup;
comparing a generation counter value stored in a generation counter field of the previous version of the changed metadata record and a generation counter value stored in a generation counter field of the current version of the changed metadata record to determine whether or not the generation counter values match;
if the generation counter values match, determining that the file object associated with the current version of the changed metadata record has been modified and recording, in the journal, information about the file object having been modified; and
if the generation counter values do not match, determining that the file object associated with the previous version of the changed metadata record has been deleted and recording, in the journal, information about the file object having been deleted.
12. The computer program product of claim 11 wherein the detecting that the event of a first type has occurred further comprises:
reading an entry for a file object in the previous version of the changed metadata record, wherein an in-use flag field in the previous version of the changed metadata record comprises a value indicating that the previous version of the changed metadata record was not in-use as of the previous backup;
reading an entry for a file object in the current version of the changed metadata record, wherein an in-use flag field in the current version of the changed metadata record comprises a value indicating that the current version of the changed metadata record was in-use as of the current backup;
determining that the file object associated with the current version of the changed metadata record is new; and
recording, in the journal, information about the file object having been newly created.
13. The computer program product of claim 11 wherein the detecting that an event of a first type has occurred further comprises:
reading an entry for a file object in the previous version of the changed metadata record, wherein an in-use flag field in the previous version of the changed metadata record comprises a value indicating that the previous version of the changed metadata record was in-use as of the previous backup;
reading an entry for a file object in the current version of the changed metadata record, wherein an in-use flag field in the current version of the changed metadata record comprises a value indicating that the current version of the changed metadata record was not in-use as of the current backup;
determining that the file object associated with the previous version of the changed metadata record was deleted; and
recording, in the journal, information about the file object having been deleted.
14. The computer program product of claim 11 wherein the detecting that the event of a second type has occurred further comprises:
reading an entry for a file object in the previous version of the changed metadata record, wherein an in-use flag field in the previous version of the changed metadata record comprises a value indicating that the previous version of the changed metadata record was not in-use as of the previous backup;
reading an entry for a file object in the current version of the changed metadata record, wherein an in-use flag field in the current version of the changed metadata record comprises a value indicating that the current version of the changed metadata record was not in-use as of the current backup;
determining that the file object associated with the current version of the changed metadata record was created and deleted between the previous and current backups of the volume; and
not recording, in the journal, information about the file object having been created and deleted between the previous and current backups.
15. The computer program product of claim 11 wherein all the blocks of the volume that have changed since the previous backup are backed up during the current backup before the intersecting.