Fragmentation mitigation in synthetic full backups
Various methods, systems, and processes for mitigating fragmentation in synthetic full backups are disclosed. One or more storage units out of multiple storage units are identified. The multiple storage units include one or more new storage units or one or more existing storage units. The multiple storage units are accessed and a determination is made as to whether the one or more storage units out of the multiple storage units meet a threshold. The threshold is a measure of a characteristic of those one or more storage units. If one or more storage units meet the threshold, those one or more storage units are included in a backup stream, and the backup stream is sent to a backup server.
1. A computer-implemented method of performing a backup operation comprising:
identifying one or more extents of data among a plurality of extents of data, wherein
the one or more extents of data are generated by a virtual machine executing on a hypervisor server,
the identification is performed by a proxy server using a changed block tracker maintained by the hypervisor server, and
the plurality of extents of data comprise one or more new extents of data or one or more existing extents of data;
accessing the plurality of extents of data;
determining whether the one or more extents of data collectively comprise at least a predetermined threshold quantity of data;
determining whether backing up the one or more extents of data will not exceed a predetermined threshold level of fragmentation;
in response to determining that the one or more extents of data collectively comprise at least a predetermined threshold quantity of data and that backing up the one or more extents of data will not exceed the predetermined threshold level of fragmentation, generating a backup stream by including, in the backup stream, the one or more extents of data that collectively comprise at least the predetermined threshold quantity of data; and
sending the backup stream to a backup server as part of the backup operation.
2. The method of claim 1 , further comprising:
obtaining the one or more extents of data from a client; and
identifying the one or more extents of data obtained from the client using a change tracking log.
3. The method of claim 2 , further comprising:
determining whether the one or more new extents of data meet a new size threshold;
including the one or more new extents of data that meet the new size threshold in the backup stream;
determining whether the one or more existing extents of data meet an existing size threshold; and
including the one or more existing extents of data that meet the existing size threshold in the backup stream.
4. The method of claim 3 , further comprising:
in response to determining that the one or more existing extents of data do not meet the existing size threshold, waiting for the client to generate additional new data prior to including the one or more existing extents of data in the backup stream.
5. The method of claim 4 , further comprising:
in response to determining that the one or more new extents of data do not meet the new size threshold, waiting for the client to generate the additional new data prior to including the one or more new extents of data in the backup stream.
6. The method of claim 1 , further comprising:
determining whether a span of extents of data among the plurality of extents of data comprises more new extents of data than existing extents of data; and
in response to determining that the span of extents of data comprises more new extents of data than existing extents of data, including the span of extents of data in the backup stream.
7. The method of claim 6 , further comprising:
in response to determining that the plurality of extents of data comprise more new extents of data than existing extents of data, including the span of extents of data in the backup stream.
8. The method of claim 7 , further comprising:
in response to determining that the plurality of extents of data comprise more existing extents of data than new extents of data, limiting a frequency with which the one or more existing extents of data are included in the backup stream.
9. The method of claim 1 , further comprising:
in response to determining that the backup stream is stored by the backup server at a storage location, calculating the predetermined threshold quantity of data, wherein
the predetermined threshold quantity of data is calculated based, at least in part, by determining whether including the one or more extents of data in the backup stream will cause fragmentation.
10. The method of claim 3 , further comprising:
in response to determining that the one or more new extents of data do not meet the new size threshold, merging two or more new extents of data into a merged extent of data; and
including the merged extent of data in the backup stream.
11. A system comprising:
one or more hardware processors; and
a memory coupled to the one or more hardware processors, wherein the memory stores program instructions executable by the one or more hardware processors to:
identify one or more extents of data among a plurality of extents of data, wherein
the one or more extents of data are generated by a virtual machine executing on a hypervisor server,
identifying the one or more extents is performed by a proxy server using a changed block tracker maintained by the hypervisor server, and
the plurality of extents of data comprise one or more new extents of data or one or more existing extents of data;
access the plurality of extents of data;
determine whether the one or more extents of data collectively comprise at least a predetermined threshold quantity of data;
determine whether backing up the one or more extents of data will not exceed a predetermined threshold level of fragmentation;
in response to determining that the one or more extents of data collectively comprise at least a predetermined threshold quantity of data and that backing up the one or more extents of data will not exceed the predetermined threshold level of fragmentation, generate a backup stream by including, in the backup stream, the one or more extents of data that collectively comprise at least the predetermined threshold quantity of data; and
send the backup stream to a backup server as part of a backup operation.
12. The system of claim 11 , wherein
identifying the one or more extents, accessing the plurality of extents of data, determining whether the one or more extends of data collectively comprise at least a predetermined threshold quantity of data, the including, and sending the backup stream is performed by an accelerator,
the accelerator is executed on the proxy server, wherein
the proxy server is communicatively coupled to the hypervisor server, and
the proxy server also comprises a deduplication engine, and
the backup server is a media server, wherein
the media server comprises a synthesis engine, a backup engine, and a storage device.
13. The system of claim 12 , wherein
the one or more extents of data of the plurality of extents of data are generated by a virtual machine executing on the hypervisor server,
a metadata module tracks whether the one or more extents of data have changed using a changed block tracker, and
the change block tracker is maintained on the hypervisor server.
14. The system of 13 , wherein the program instructions are further executable to:
identifying the one or more new extents of data of the plurality of extents of data using the accelerator;
sending the one or more new extents of data from the virtual machine to the proxy server;
retrieving a state file from a master server, wherein
the state file comprises information about each extent of data stored on a virtual disk associated with the virtual machine, and
the state file information indicates whether the one or more new extents of data have already been backed up to the storage device as part of a previous full backup operation or as part of a previous incremental backup operation; and
in response to determining that the one or more new extents of data or the one or more existing extents of data meet the predetermined threshold quantity of data, including the one or more new extents of data or the one or more existing extents of data in the backup stream.
15. The system of claim 14 , wherein the program instructions are further executable to:
sending the backup stream and the state file information to the deduplication engine;
deduplicating the backup stream;
generating, using the synthesis engine, a synthetic full backup on the media server;
updating the state file;
transferring the state file to the master server;
transferring catalog information for the synthetic full backup to the master server to be stored in a catalog, wherein
the catalog comprises location information about the synthetic full backup; and
storing the synthetic full backup on the media server.
16. A non-transitory computer readable storage medium comprising program instructions executable to:
identify one or more extents of data among a plurality of extents of data, wherein
the one or more extents of data are generated by a virtual machine executing on a hypervisor server,
identifying the one or more extends is performed by a proxy server using a changed block tracker maintained by the hypervisor server, and
the plurality of extents of data comprise one or more new extents of data or one or more existing extents of data;
access the plurality of extents of data;
determine whether the one or more extents of data collectively comprise at least a predetermined threshold quantity of data;
determine whether backing up the one or more extents of data will not exceed a predetermined threshold level of fragmentation;
in response to determining that the one or more extents of data collectively comprise at least a predetermined threshold quantity of data and that backing up the one or more extents of data will not exceed the predetermined threshold level of fragmentation,
generate a backup stream by including, in the backup stream, the one or more extents of data that collectively comprise at least the predetermined threshold quantity of data; and
send the backup stream to a backup server as part of a backup operation.
17. The non-transitory computer readable storage medium of claim 16 , comprising:
obtaining the one or more extents of data from a client;
identifying the one or more extents of data obtained from the client using a change tracking log;
determining whether the one or more new extents of data meet a new size threshold;
including the one or more new extents of data that meet the new size threshold in the backup stream;
determining whether the one or more existing extents of data meet an existing size threshold; and
including the one or more existing extents of data that meet the existing size threshold in the backup stream.
18. The non-transitory computer readable storage medium of claim 17 , further comprising:
in response to determining that the one or more existing extents of data do not meet the existing size threshold, waiting for the client to generate additional new data prior to including the one or more existing extents of data in the backup stream; and
in response to determining that the one or more new extents of data do not meet the new size threshold, waiting for the client to generate the additional new data prior to including the one or more new extents of data in the backup stream.
19. The non-transitory computer readable storage medium of claim 16 , wherein
identifying the one or more extents, accessing the plurality of extents of data, determining whether the one or more extents of data collectively comprise at least a predetermined threshold quantity of data, the including, and sending the backup stream is performed by an accelerator,
the accelerator is executed on a proxy server, wherein
the proxy server is communicatively coupled to a hypervisor server, and
the proxy server also comprises a deduplication engine,
the backup server is a media server, wherein
the media server comprises a synthesis engine, a backup engine, and a storage device,
the one or more extents of data among the plurality of extents of data are generated by a virtual machine executing on the hypervisor server,
a metadata module tracks whether the one or more extents of data have changed using a changed block tracker, and
the change block tracker is maintained on the hypervisor server.
20. The non-transitory computer readable storage medium of claim 19 , comprising:
identifying the one or more new extents of data of the plurality of extents of data using the accelerator;
sending the one or more new extents of data from the virtual machine to the proxy server;
retrieving a state file from a master server, wherein
the state file comprises information about each of the extents of data stored on a virtual disk associated with the virtual machine, and
the state file information indicates whether the one or more new extents of data have already been backed up to the storage device as part of a previous full backup operation or as part of a previous incremental backup operation;
in response to determining that the one or more new extents of data or the one or more existing extents of data meet the predetermined threshold quantity of data, including the one or more new extents of data or the one or more existing extents of data in the backup stream;
sending the backup stream and the state file information to the deduplication engine;
deduplicating the backup stream;
generating, using the synthesis engine, a synthetic full backup on the media server;
updating the state file;
transferring the state file to the master server;
transferring catalog information for the synthetic full backup to the master server to be stored in a catalog, wherein
the catalog comprises location information about the synthetic full backup, and
storing the synthetic full backup on the media server.