Optimizing incremental backup for clients in a dedupe cluster to provide faster backup windows with high dedupe and minimal overhead
An intelligent method of handling incremental backups concurrent with load balancing movement. The file system uses placement tags, incremental backup requests and capacity balancing data movement to make intelligent decision to avoid affecting any backup windows for clients or backup apps. The file system tracks capacity balancing file movements inside the cluster. When switching locations of files in a cluster from one node to another, it is performed as an atomic change of switching inode attributes by the file system after the contents of the file have been copied over to the new node. During the file movement for capacity balancing, the file system handles requests for full backups differently than requests for incremental backups. The file system continues to handle virtual systhesis and fastcopy requests on the node that hosts the previous backup to ensure that the incremental backup succeeds with the expected smaller backup window from the client.
1. A system comprising:
one or more processors; and
a non-transitory computer readable medium storing a plurality of instructions, which when executed, cause the one or more processors to:
receive first data to be backed up from a client system as part of a backup process, the first data associated with a first data source ID of the client system;
determine a selected node for storage of the first data to be backed up according to the first data source ID of the client system;
determine that a fullness threshold has been reached on the selected node;
determine that the first data to be backed up is a full backup;
initiate a move of the full backup to an available node and send instructions to the client system to initiate a next full backup having the first data source ID on the available node, wherein the initiating the move of the full backup to the available node is based on the determination that the fullness threshold has been reached on the selected node and the determination that the first data to be backed up is the full backup;
receive second data to be backed up from the client system as part of the backup process, the second data associated with a second data source ID of the client system;
determine the selected node for storage of the second data to be backed up according to the second data source ID of the client system;
determine that the second data to be backed up is an incremental backup; and
initiate a move of a prior full backup corresponding to the second data source ID to the available node and initiate the incremental backup of the received second data on the selected node, wherein the initiating the move of the prior full backup is based on the determination that the fullness threshold has been reached on the selected node and the determination that the second data to be backed up is the incremental backup.
2. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
once the incremental backup of the received second data on the selected node is completed, move the received second data of the incremental backup to the available node.
3. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
responsive to determining that the fullness threshold has been reached on the selected node and the second data to be backed up is the incremental backup, determine that a move of a prior full backup corresponding to the second data source ID to the available node has already been committed; and
initiate the incremental backup on the available node based on the determination that the move of the prior full backup corresponding to the second data source ID to the available node has already been committed.
4. A computer implemented method comprising:
receiving first data to be backed up from a client system as part of a backup process, the first data associated with a first data source ID of the client system;
determining a selected node for storage of the first data to be backed up according to the first data source ID of the client system;
determining that a fullness threshold has been reached on the selected node;
determining that the first data to be backed up is a full backup;
initiating a move of the full backup to an available node and sending instructions to the client system to initiate a next full backup having the first data source ID on the available node, wherein the initiating the move of the full backup to the available node is based on the determination that the fullness threshold has been reached on the selected node and the determination that the first data to be backed up is the full backup;
receiving second data to be backed up from the client system as part of the backup process, the second data associated with a second data source ID of the client system;
determining the selected node for storage of the second data to be backed up according to the second data source ID of the client system;
determining that the second data to be backed up is an incremental backup; and
initiating a move of a prior full backup corresponding to the second data source ID to the available node and initiating the incremental backup of the received second data on the selected node, wherein the initiating the move of the prior full backup is based on the determination that the fullness threshold has been reached on the selected node and the determination that the second data to be backed up is the incremental backup.
5. The method of claim 4 , further comprising: once the incremental backup of the received second data on the selected node is completed, moving the received second data of the incremental backup to the available node.
6. The method of claim 4 , further comprising: responsive to determining that the fullness threshold has been reached on the selected node and the second data to be backed up is the incremental backup, determining that a move of a prior full backup corresponding to the second data source ID to the available node has already been committed; and
initiating the incremental backup on the available node based on the determination that the move of the prior full backup corresponding to the second data source ID to the available node has already been committed.
7. A computer implemented method comprising non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
receive first data to be backed up from a client system as part of a backup process, the first data associated with a first data source ID of the client system;
determine a selected node for storage of the first data to be backed up according to the first data source ID of the client system;
determine that a fullness threshold has been reached on the selected node;
determine that the first data to be backed up is a full backup;
initiate a move of the full backup to an available node and sending instructions to the client system to initiate a next full backup having the first data source ID on the available node, wherein the initiating the move of the full backup to the available node is based on the determination that the fullness threshold has been reached on the selected node and the determination that the first data to be backed up is the full backup;
receive second data to be backed up from the client system as part of the backup process, the second data associated with a second data source ID of the client system;
determine the selected node for storage of the second data to be backed up according to the second data source ID of the client system;
determine that the second data to be backed up is an incremental backup; and
initiate a move of a prior full backup corresponding to the second data source ID to the available node and initiate the incremental backup of the received second data on the selected node, wherein the initiating the move of the prior full backup is based on the determination that the fullness threshold has been reached on the selected node and the determination that the second data to be backed up is the incremental backup.
8. The method non-transitory computer-readable medium of claim 7 , further comprising deleting wherein the program code further includes instructions to delete all data corresponding to the first data source ID from the selected node.