System to catalog and search point-in-time instances of a file system
View Patent ↗A system to catalog and search point-in-tine instances of a file system is disclosed. A catalog engine takes backups of file data generated by a storage system and catalogs the backups of file data into a searchable catalog of independent metadata records. The metadata is represented by baseline structure and delta files.
1. A computer implemented method, the method comprising:
monitoring, by a catalog engine, the creation or deletion of point-in-time backups from each of a plurality of N different file systems in which independent processes are utilized to acquire data from each of the N different file systems;
determining, for each of the N different file systems, differences between a set of backups known to the catalog engine and to backups known by each of N different file systems;
building a central catalog of directory metadata based on the determined differences between the set of backups known to the catalog engine and the backups known by each of the N different file systems, the central catalog of directory metadata including a set of independent backup catalogs representing the N different file systems in which each individual backup catalog represents point-in-time backups of a respective one of the N different file systems;
wherein the central catalog is organized in metadata records and is tangibly embodied in the at least one tangible computer readable medium organized in records arranged as baseline files and deltas, and the metadata records for the N different file systems are stored as N independent backup catalog structures including pointers to files and directories of the plurality of different file systems;
wherein each record for an individual backup file catalog has a file structure that includes:
an index of hashes of directory names for the N different file systems;
a breadth first ordered set of entries or change records that have happened between a backup and its next younger sibling, with a pointer to the directory metadata;
a breadth first ordered set of directory moves divided into compressed pages;
a sorted skip list of file and directory names with a pointer to the entry record, wherein each entry in the skip list of file and directory names is hashed and stored in the index of hashes; and
a breadth first ordered set of directory paths, divided into compressed pages.
2. The method of claim 1 wherein the metadata records are stored in a logical sequence that is in a breadth first manner of a hierarchical structure.
3. The method of claim 1 , wherein the directory metadata that is acquired for cataloging an individual backup file catalog includes one or more of: listing of changed files, a listing of changed file system pointers or inodes, a list of changed storage blocks, a complete listing of all of the files on an individual file system.
4. The method of claim 1 , wherein in the event that a delta file is deleted for a data structure of a backup file catalog, a rollup is performed in the cataloged deltas to adapt the catalog for the individual file system.
5. The method of claim 1 , further comprising processing the directory metadata acquired from a storage system and to correlate primitive data acquired into higher level catalog engine events into a homogeneous set of record types.
6. The method of claim 1 , further comprising searching the catalog to identify metadata for backup files in response to a query, including performing at least a portion of the searching via hashed information of the file structure of the individual backup file catalog.
7. A computer readable media product comprising a non-transitory computer readable medium causing a processor to implement any of the methods of claim 1 , 2 , 3 , 4 , 5 , or 6 .
8. A catalog engine comprising a processor and a memory, the catalog engine configured to implement any of the methods of claim 1 , 2 , 3 , 4 , 5 , or 6 to catalog snapshot data generated by a separate backup system.
9. A computer implemented method, the method comprising:
monitoring, by a catalog engine, the creation or deletion of point-in-time backups from each of a plurality of N different file systems in which independent processes are utilized to acquire data from each of the N different file systems;
determining, for each of the N different file systems, differences between a set of backups known to the catalog engine and to backups known by each of N different file systems;
building a central catalog of directory metadata based on the determined differences between the set of backups known to the catalog engine and the backups known by each of the N different file systems, the central catalog of directory metadata including a set of independent backup catalogs representing the N different file systems in which each individual backup catalog represents point-in-time backups of a respective one of the N different file systems, wherein the central catalog is organized in metadata records and is tangibly embodied in the at least one tangible computer readable medium organized in records arranged as baseline files and deltas, and the metadata records for the N different file systems are stored as N independent backup catalog structures including pointers to files and directories of the plurality of different file systems and each record for an individual backup file catalog has a file structure that includes:
1) index of hashes of directory names for the N different file systems;
2) a breadth first ordered set of entries or change records that have happened between a backup and its next younger sibling, with a pointer to the directory metadata;
3) a breadth first ordered set of directory moves divided into compressed pages;
4) a sorted skip list of file and directory names with a pointer to the entry record, wherein each entry in the skip list of file and directory names is hashed and stored in the index of hashes; and
5) a breadth first ordered set of directory paths, divided into compressed pages;
independently adding a new backup catalog to the central catalog;
independently deleting a backup catalog from the central catalog;
deleting a delta file for a data structure of a backup catalog by performing a rollup in the cataloged deltas to adapt the backup catalog for the individual file system; and
searching the catalog to identify metadata for backup files in response to a query, including performing at least a portion of the searching via hashed information of the file structure of the individual backup file catalog.