Metadata management for scaled and high density backup environments
Embodiments for managing metadata in a high density network attached storage system. A change file list is obtained though external or internal crawler processes. A slicer creates multiple slices of the entire fileshare to enable parallel backup of the slices. A backup engine creates a separate backup containers and separate metadata files for each slice. The individual metadata files are combined into a single consolidated metadata file, and the individual metadata files are then deleted. The consolidated metadata file contains all of the elements comprising the entire fileshare, and a corresponding container ID is also added to each entry of the file. Another table contains details of the sub-assets and their backup properties. A backup agent obtains the backup IDs for the individual sub-assets, using a job ID field for a backup query operating on the single consolidated metadata file.
1. A computer-implemented method of managing metadata for backup data in a network attached storage (NAS) fileshare, comprising:
obtaining a change file list (CFL) listing only files that are changed between two different snapshot backups, and obtained by one of an external source providing the CFL or a crawler process crawling the fileshare, to determine changes in files between successive backup operations, wherein the changes in a file comprise at least one of: newly added data, modified data, and deleted data;
creating, by a processor-based backup engine, multiple slices for parallel backup of the files, wherein each slice comprises one or more files of the NAS fileshare;
creating separate backup containers and separate metadata files for each slice;
combining the separate metadata files into a single consolidated metadata file;
defining file details and backup properties in a backup table;
processing, by the processor-based backup engine, a backup query using the consolidated metadata file and backup table;
responding to the backup query through an interface of the backup engine by returning a unique backup ID for each file of the files;
generating the multiple slices by an NAS agent for the backup;
performing an incremental backup using the CFL;
obtaining, by the NAS agent the CFL for the entire fileshare;
using the consolidated metadata file from a last backup to identify the changes in each file to generate storage buckets therefor;
passing the generated storage buckets to a backup engine; and
using the consolidated metadata file again for identifying unchanged files for synthesis from the last backup to get a location of the unchanged files in the previous backup container for synthesizing into a new container.
2. The method of claim 1 further comprising dynamically redistributing the slices of the fileshare to create each of the multiple slices of approximately a same size.
3. The method of claim 2 wherein the redistributing comprises:
re-slicing a slice that is at least double an average slice size; and
combining together slices that are at least half the average slice size.
4. The method of claim 1 further comprising performing an incremental backup obtained by crawling the fileshare.
5. The method of claim 4 further comprising:
performing the incremental backup of the files on each slice;
passing, from the NAS agent to a file system agent, the consolidated metadata file of a previous backup;
passing, from the NAS agent to the file system agent, a bucket of files to be backed up;
looking up, in the consolidated metadata file only, files in each slice; and
updating and inserting records of changed files in the consolidated metadata file either in the individual metadata file prior to combining in the consolidated metadata file or in the consolidated metadata file after the combining.
6. The method of claim 1 further comprising storing the consolidated metadata file as a catalog separate from one or more catalogs of data for the fileshare.
7. A computer-implemented method of managing metadata for backup data in a network attached storage (NAS) fileshare, comprising:
defining, in a hardware component of a processor-based backup engine, a one-to-one mapping between a backup asset of a plurality of backup assets and its corresponding metadata in a consolidated metadata file;
mapping all the constituent containers of the plurality of backup assets for reference from the consolidated metadata file;
associating, in a backup table used by the backup engine, each backup asset with a backup job identifier of a last backup operation performed on the backup asset;
processing, through an interface of the backup engine, backup queries using the consolidated metadata file and the backup table,
obtaining a change file list (CFL) listing only files that are changed between two different snapshot backups; and
creating multiple slices for parallel backup of the files, wherein each slice includes a sub-asset comprising one or more files of the NAS fileshare, wherein the CFL is obtained by one of: an external source providing the CFL, or a crawler of the backup engine crawling the fileshare to determine changes in files between successive backup operations, wherein the changes in a file comprise at least one of: newly added data, modified data, and deleted data, the NAS agent storing the consolidated metadata file as a catalog separate from one or more catalogs of data for the fileshare, and further wherein the NAS agent further obtains the CFL for the entire fileshare, uses the consolidated metadata file from a last backup to identify the changes in each file to generate buckets therefor, passes the generated buckets to the backup engine, and uses the consolidated metadata file again to identify unchanged files for synthesis from the last backup to get a location of the unchanged files in the previous backup container for synthesizing into a new container.
8. The method of claim 7 wherein the CFL is obtained by one of: an external source providing the CFL, or crawling the fileshare to determine changes in files between successive backup operations, wherein the changes in a file comprise at least one of: newly added data, modified data, and deleted data.
9. The method of claim 8 further comprising dynamically redistributing the slices of the fileshare to create slices of approximately same size by re-slicing a slice that is at least double an average slice size, or combining together slices that are at least half the average slice size.
10. The method of claim 9 further comprising performing an incremental backup using one of the CFL as provided by the external source, or obtained by a crawler process crawling the fileshare.
11. The method of claim 10 further comprising:
obtaining, by the NAS agent, the CFL for the entire fileshare;
using the consolidated metadata file from a last backup to identify the changes in each file to generate buckets therefor;
passing the generated buckets to a backup engine; and
using the consolidated metadata file again for identifying unchanged files for synthesis from the last backup to get a location of the unchanged files in the previous backup container for synthesizing into a new.
12. The method of claim 10 further comprising:
performing the incremental backup of the files on each slice;
passing, from the NAS agent to a file system agent, the consolidated metadata file of a previous backup;
passing, from the NAS agent to the file system agent, a bucket of files to be backed up;
looking up, in the consolidated metadata file only, files in each slice; and
updating and inserting records of changed files in the consolidated metadata file either in the individual metadata file prior to combining in the consolidated metadata file or in the consolidated metadata file after the combining.
13. A system for managing metadata for backup data in a network attached storage (NAS) fileshare, comprising:
a hardware component comprising an NAS agent obtaining a change file list (CFL) listing only files that are changed between two different snapshot backups, creating multiple slices for parallel backup of the files, wherein each slice comprises one or more files of the NAS fileshare, creating separate backup containers and separate metadata files for each slice, combining the separate metadata files into a single consolidated metadata file, defining file details and backup properties in a backup table; and
a hardware-based backup engine component processing backup queries using the consolidated metadata file and backup table, responding to a backup query by returning a unique backup ID for each file of the files, and generating storage buckets by an NAS agent for the backup, wherein the CFL is obtained by one of: an external source providing the CFL, or a crawler of the backup engine crawling the fileshare to determine changes in files between successive backup operations, wherein the changes in a file comprise at least one of: newly added data, modified data, and deleted data, the NAS agent storing the consolidated metadata file as a catalog separate from one or more catalogs of data for the fileshare, and further wherein the NAS agent further obtains the CFL for the entire fileshare, uses the consolidated metadata file from a last backup to identify the changes in each file to generate buckets therefor, passes the generated buckets to the backup engine, and uses the consolidated metadata file again to identify unchanged files for synthesis from the last backup to get a location of the unchanged files in the previous backup container for synthesizing into a new container.