Systems and methods for efficiently backing up large datasets
A computer-implemented method for efficiently backing up large datasets may include (i) identifying data on an application server to be deduplicated by a deduplication server and then stored on a backup server, (ii) dividing the data into subsets, and (iii) for each subset of data subsequent to an initial subset of data, (a) transferring the subset of data to the deduplication server in response to detecting that a previous subset of data has completed transfer to the deduplication server, (b) deduplicating the subset of data in response to detecting that the previous subset of data has completed deduplication, and (c) transferring a deduplicated version of the subset of data to the backup server in response to detecting that the subset of data has completed deduplication and the previous subset of data has completed transfer to the backup server. Various other methods, systems, and computer-readable media are also disclosed.
1. A computer-implemented method for efficiently backing up large datasets, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:
identifying a set of data on an application server, the application server comprising an application data storage, to be deduplicated by a deduplication server, the deduplication server comprising a deduplication data storage, and stored on a backup server;
dividing the set of data on the application server into a plurality of subsets of data; and
for each subset of data subsequent to an initial subset of data:
in response to detecting that a previous subset of data has completed transfer to the deduplication data storage, transferring the subset of data from the application data storage to the deduplication data storage while the deduplication server performs deduplication of the previous subset of data;
in response to detecting that the previous subset of data has completed deduplication by the deduplication server and that the subset of data has completed transfer to the deduplication data storage, deduplicating, by the deduplication server, the subset of data while the deduplication server transfers the previous subset of data to the backup server; and
in response to detecting that the previous subset of data has completed transfer to the backup server and that the subset of data has completed deduplication at the deduplication server, transferring a deduplicated version of the subset of data to the backup server.
2. The computer-implemented method of claim 1 , wherein dividing the set of data on the application server into the plurality of subsets of data comprises dividing the set of data such that each subset is a unique dataset subject to deduplication independent of each other subset.
3. The computer-implemented method of claim 1 , wherein dividing the set of data on the application server into the plurality of subsets of data comprises dividing the set of data such that each subset occupies less space in storage than an amount of space in storage available on the deduplication server.
4. The computer-implemented method of claim 1 , wherein the set of data comprises a set of columns of a database on the application server.
5. The computer-implemented method of claim 1 , wherein the application server comprises a cluster of application servers.
6. The computer-implemented method of claim 1 , wherein the deduplication server comprises a cluster of deduplication servers.
7. The computer-implemented method of claim 1 , wherein the application server comprises a greater amount of storage than the deduplication server.
8. The computer-implemented method of claim 1 , wherein the deduplication server comprises insufficient storage to store the set of data from the application server.
9. A system for efficiently backing up large datasets, the system comprising:
an identification module, stored in memory, that identifies a set of data on an application server, the application server comprising an application data storage, to be deduplicated by a deduplication server, the deduplication server comprising a deduplication data storage, and stored on a backup server;
a division module, stored in memory, that divides the set of data on the application server into a plurality of subsets of data;
a transfer module, stored in memory, that, for each subset of data subsequent to an initial subset of data, in response to detecting that a previous subset of data has completed transfer to the deduplication data storage, transfers the subset of data from the application data storage to the deduplication data storage while the deduplication server performs deduplication of the previous subset of data;
a deduplication module, stored in memory, that, for each subset of data subsequent to the initial subset of data, in response to detecting that the previous subset of data has completed deduplication by the deduplication server and that the subset of data has completed transfer to the deduplication data storage, deduplicates, by the deduplication server, the subset of data while the deduplication server transfers the previous subset of data to the backup server;
a storage module, stored in memory, that, for each subset of data subsequent to the initial subset of data, in response to detecting that the previous subset of data has completed transfer to the backup server and that the subset of data has completed deduplication at the deduplication server, transfers a deduplicated version of the subset of data to the backup server; and
at least one physical processor configured to execute the identification module, the division module, the transfer module, the deduplication module, and the storage module.
10. The system of claim 9 , wherein the division module divides the set of data on the application server into the plurality of subsets of data by dividing the set of data such that each subset is a unique dataset subject to deduplication independent of each other subset.
11. The system of claim 9 , wherein the division module divides the set of data on the application server into the plurality of subsets of data by dividing the set of data such that each subset occupies less space in storage than an amount of space in storage available on the deduplication server.
12. The system of claim 9 , wherein the set of data comprises a set of columns of a database on the application server.
13. The system of claim 9 , wherein the application server comprises a cluster of application servers.
14. The system of claim 9 , wherein the deduplication server comprises a cluster of deduplication servers.
15. The system of claim 9 , wherein the application server comprises a greater amount of storage than the deduplication server.
16. The system of claim 9 , wherein the deduplication server comprises insufficient storage to store the set of data from the application server.
17. A non-transitory computer-readable medium comprising one or more computer-readable instructions that, when executed by at least one processor of a computing device, cause the computing device to:
identify a set of data on an application server, the application server comprising an application data storage, to be deduplicated by a deduplication server, the deduplication server comprising a deduplication data storage, and stored on a backup server;
divide the set of data on the application server into a plurality of subsets of data; and
for each subset of data subsequent to an initial subset of data:
in response to detecting that a previous subset of data has completed transfer to the deduplication data storage, transfer the subset of data from the application data storage to the deduplication data storage while the deduplication server performs deduplication of the previous subset of data;
in response to detecting that the previous subset of data has completed deduplication by the deduplication server and that the subset of data has completed transfer to the deduplication data storage, deduplicate, by the deduplication server, the subset of data while the deduplication server transfers the previous subset of data to the backup server; and
in response to detecting that the previous subset of data has completed transfer to the backup server and that the subset of data has completed deduplication at the deduplication server, transfer a deduplicated version of the subset of data to the backup server.
18. The non-transitory computer-readable medium of claim 17 , wherein the one or more computer-readable instructions cause the computing device to divide the set of data on the application server into the plurality of subsets of data by dividing the set of data such that each subset is a unique dataset subject to deduplication independent of each other subset.
19. The non-transitory computer-readable medium of claim 17 , wherein the one or more computer-readable instructions cause the computing device to divide the set of data on the application server into the plurality of subsets of data by dividing the set of data such that each subset occupies less space in storage than an amount of space in storage available on the deduplication server.
20. The non-transitory computer-readable medium of claim 17 , wherein the set of data comprises a set of columns of a database on the application server.