Intelligent block slicing in network attached storage for data protection
Embodiments are described for storing array snapshots of a block-based system in networked storage, such as an NAS or SAN device. A system and process determines a size of the block device and splits it into a number of slices based on the size of the block device, with each slice comprising slice data. The slice data is written to protection storage either directly or through temporary buffer memory. The slice number and memory location for the slice data is stored in a key/value map, where the key comprises the slice number, and the value comprises the location. Backup agents are deployed to back up the slice data for each slice of the block device to the networked storage, and the slice data is stored as array snapshots in the network storage.
1. A computer-implemented method of storing array snapshots of a block device in networked storage and a file system in networked attached storage (NAS) device, comprising:
determining an organization of the file system comprising directories and files in a defined hierarchy;
determining a size of each file and a number of files in each directory;
first slicing the file system into a number of file slices according to a combined slicing method comprising file size-based slicing for large file systems, and file count-based slicing for dense file systems with each file slice comprising file slice data, wherein the large file systems are used in metropolitan-area network (MAN) scale networks, and the dense file systems comprise millions of files;
determining a size of the block device accessed either directly or as a read-only copy;
second slicing the block device into a number of block slices based on the size of the block device, with each block slice comprising block slice data;
writing both the file slice data and block slice data to protection storage;
deploying backup agents to back up the slice data for each slice of the block device and file system to a network attached storage (NAS) device, wherein the backup agents are installed on an as-needed basis on one or more proxy hosts automatically with proxy host deployment, wherein each proxy of the proxy hosts acts as an execution place embodied in a Virtual Machine (VM) for requests sent by the client to access resources of the networked storage;
storing a corresponding slice number for each of the file slice data and block slice data in a key/value map by a backup agent, where the key comprises the slice number, and the value comprises the protection storage location of the corresponding file or block slice, and wherein the corresponding slice number is stored in temporary storage and separately from the slice data to prevent modification of the slice data;
reading, by the backup agent, each file and block slice for storage in protection memory and creation of the key/value map by the backup agent; and
storing the backed up slices as array snapshots in the NAS device.
2. The method of claim 1 further comprising storing the key/value map separately from the slice data.
3. The method of claim 2 further comprising first storing the slice data in a temporary memory location as a buffer, and writing the slice data to the protection storage after the buffer of the temporary memory location is full.
4. The method of claim 3 wherein the networked storage comprises one of a network attached storage (NAS) device or a storage area network (SAN) device.
5. The method of claim 4 wherein the block device comprises a storage device that supports reading and writing data in one of fixed-size blocks or variable-sized blocks.
6. The method of claim 5 wherein the block device comprises a file defined by an operating system, and wherein the file is split into smaller file slices.
7. The method of claim 1 wherein the backup operation comprises a full backup followed by one or more incremental backups, and wherein the method further comprises:
storing, in a catalog, slicing results after each of the full and one or more incremental backups;
determining, prior to each incremental backup, slicing results of a next previous full or incremental backup; and
re-slicing the file system based on slice count.
8. The method of claim 1 wherein the backup agents are deployed as proxies in a client of a data management system in relation to the number of file and block slices.
9. The method of claim 8 wherein the deployment of backup agents is performed by one of an automated process based on the number of slices, and a pre-defined manual process.
10. The method of claim 9 wherein the storage system comprises a deduplication storage system, and wherein the backup is performed between nodes utilizing Data Domain (DD) protection storage, comprising a random access storage device implemented through one of a disk-based storage device, a redundant array of independent disks, or a virtual disk (VHD).
11. A system for storing array snapshots of a file system in networked storage, comprising a hardware processor configured to execute software program code to perform a method of storing array snapshots of a block device in the networked storage and a file system in networked attached storage (NAS) device, comprising:
determining an organization of the file system comprising directories and files in a defined hierarchy;
determining a size of each file and a number of files in each directory;
first slicing the file system into a number of file slices according to a combined slicing method comprising file size-based slicing for large file systems, and file count-based slicing for dense file systems with each file slice comprising file slice data, wherein the large file systems are used in metropolitan-area network (MAN) scale networks, and the dense file systems comprise millions of files;
determining a size of the block device accessed either directly or as a read-only copy;
second slicing the block device into a number of block slices based on the size of the block device, with each block slice comprising block slice data;
writing both the file slice data and block slice data to protection storage;
deploying backup agents to back up the slice data for each slice of the block device and file system to a network attached storage (NAS) device, wherein the backup agents are installed on an as-needed basis on one or more proxy hosts automatically with proxy host deployment, wherein each proxy of the proxy hosts acts as an execution place embodied in a Virtual Machine (VM) for requests sent by the client to access resources of the networked storage;
storing a corresponding slice number for each of the file slice data and block slice data in a key/value map by a backup agent, where the key comprises the slice number, and the value comprises the protection storage location of the corresponding file or block slice, and wherein the corresponding slice number is stored in temporary storage and separately from the slice data to prevent modification of the slice data;
reading, by the backup agent, each file and block slice for storage in protection memory and creation of the key/value map by the backup agent; and
storing the backed up slices as array snapshots in the NAS device.
12. The system of claim 11 wherein the key/value map is stored separately from the slice data to accommodate incremental backups and restore operations.
13. The system of claim 12 wherein the method further comprises first storing the slice data in a temporary memory location as a buffer, and second comprises writing the slice data to the protection storage after the buffer of the temporary memory location is full.
14. The system of claim 11 wherein the block device comprises a storage device that supports reading and writing data in fixed-size blocks, and wherein the backup agents are deployed as proxies in a client of a data management system supporting the backup operation, and wherein a slicer second slicing the block is implemented as a shared library embedded in the backup agents.