AUTO SCALE BACKUP ORCHESTRATION FOR NETWORK ATTACHED STORAGE WORKLOADS
In general, embodiments of the invention relate to a method and system for backing up data. More specifically, embodiments of the invention are directed to using a scalable backup infrastructure that enables portions of the backup process to be performed in parallel. The amount of parallelism that may be implemented in the backup process may be dynamically adjusted based on customer requirements and/or limitations on the computing devices, backup storage, and/or production storage that are used to perform the backup process.
1 . A method for backing up data, the method comprising:
initiating performance of a pre-work phase on a proxy host of a plurality of proxy hosts to obtain a set of slices and copy information, wherein the pre-work phase is associated with a backup request for target data on production storage;
determining a number of jobs to generate to service the backup request based on at least one of a slice allocation threshold and a parallel processing threshold;
generating a set of jobs based on the numbers of jobs, wherein each job in the set of jobs is associated with at least one slice of the set of slices;
initiating performance of the set of jobs on at least one of a plurality of proxy hosts;
determining that performance of the set of jobs has been completed; and
initiating performance of a post-work phase on a second proxy host of the plurality of proxy hosts, wherein servicing of the backup request is complete when the post-work phase is completed.
2 . The method of claim 1 , wherein initiating performance of the pre-work phase on the proxy host comprises:
instantiating a container on the proxy host, wherein the container comprises a backup agent, which when executing in the container, generates a snapshot of the target data and logically divided the snapshot in the set of slices.
3 . The method of claim 2 , wherein the container is removed from the proxy host after the pre-work phase is completed.
4 . The method of claim 1 , wherein initiating performance of the set of jobs on the at least one of a plurality of proxy hosts comprises:
instantiating a container for a job in the set of jobs;
performing, by the container, the job, wherein performing the job comprises reading data associated with a slice of the set of slices from the production storage and writing the data associated with the slice to backup storage;
instantiating a second container for a second job in the set of jobs;
performing, by the second container, the second job, wherein performing the second job comprises reading second data associated with a second slice of the set of slices from the production storage and writing the second data associated with the second slice to backup storage.
5 . The method of claim 4 , wherein the job and the second job are performed in parallel by the container and the second container.
6 . The method of claim 4 , wherein the container performs the job using a plurality of threads, wherein the plurality of threads execute in parallel.
7 . The method of claim 1 , wherein the proxy host is a physical computing device or a logical computing device.
8 . The method of claim 1 , wherein the production storage is network attached storage that is used by a production system.
9 . The method of claim 1 , wherein the proxy host and the second proxy host is the same proxy host.
10 . The method of claim 1 , wherein the slice allocation threshold specifies a maximum number of slices that may be assigned to a container.
11 . The method of claim 1 , wherein the parallel processing threshold specifies a maximum number of concurrently executing threads that may be processing jobs associated with the target data.
12 . A system, comprising:
a data protection manager;
a proxy host and a second proxy host;
backup storage;
production storage;
wherein the data protection manager, the proxy host, the second proxy host, the backup storage and the production storage are operatively connected;
wherein the data protection manager is configured to:
initiate performance of a pre-work phase on the proxy host obtain a set of slices and copy information, wherein the pre-work phase is associated with a backup request for target data on the production storage;
determine a number of jobs to generate to service the backup request based on at least one of a slice allocation threshold and a parallel processing threshold;
generate a set of jobs based on the number of jobs, wherein each job in the set of jobs is associated with at least one slice of the set of slices;
initiate performance of the set of jobs on the proxy host and the second proxy host;
determine that performance of the set of jobs has been completed; and
initiate performance of a post-work phase on the second proxy host, wherein servicing of the backup request is complete when the post-work phase is completed.
13 . The system of claim 12 , wherein initiating performance of the pre-work phase on the proxy host comprises:
instantiating a container on the proxy host, wherein the container comprises a backup agent, which when executing in the container, generates a snapshot of the target data and logically divided the snapshot in the set of slices.
14 . The system of claim 13 , wherein the container is removed from the proxy host after the pre-work phase is completed.
15 . The system of claim 12 , wherein initiating performance of the set of jobs on the proxy host and the second proxy host:
instantiating a container on the proxy host for a job in the set of jobs;
performing, by the container, the job, wherein performing the job comprises reading data associated with a slice of the set of slices from the production storage, and writing the data associated with the slice to the backup storage;
instantiating a second container on a second proxy host for a second job in the set of jobs;
performing, by the second container, the second job, wherein performing the second job comprises reading second data associated with a second slice of the set of slices from the production storage and writing the second data associated with the second slice to the backup storage.
16 . The system of claim 15 , wherein the job and the second job are performed in parallel by the container and the second container.
17 . The system of claim 15 , wherein the container performs the job using a plurality of treads, wherein the plurality of threads execute in parallel.
18 . The system of claim 12 , wherein the proxy host is a physical computing device or a logical computing device.
19 . The system of claim 12 , wherein the production storage is network attached storage that is used by a production system.
20 . The system of claim 12 ,
wherein the slice allocation threshold specifies a maximum number of slices that may be assigned to a container, and
wherein the parallel processing threshold specifies a maximum number of concurrently executing threads that may be processing jobs associated with the target data.