Application-based packing for storing backup data to an object storage
Described is a system (and method) for application-based packing for storing backup data within an object storage. More particularly, the system may efficiently pack backup files into objects having a specialized size by accounting for the source and size of the backup file. For example, the system may determine whether backup files associated with a particular application may be stored within individual objects or packed together within a shared object. Accordingly, by accounting for such criteria the system may perform an intelligent allocation of backup data within a cloud-based object storage. The system may be implemented as part of an incremental backup infrastructure. Accordingly, when allocating the backup files amongst objects, the system may also store specialized metadata to preserve the ability to perform a reconstruction of data (e.g. synthetically) as if a full backup was performed.
1. A system comprising:
one or more processors; and
a non-transitory computer-readable medium storing a plurality of instructions, which when executed, cause the one or more processors to:
receive a request to back up files stored on a client device to an object storage;
identify, amongst the files stored on the client device, a first set of files that are each associated with a first application, and a second set of files that are each associated with a second application;
determine a backup of each file of the first set of files associated with the first application are to be stored within their own objects;
identify one or more combination of files from at least some of the second set of files that when grouped together are within a range of a predetermined object size; and
perform a backup of the files stored on the client device by storing the backup of each file of the first set of files as an individual object on the object storage, in response to determining that the first set of files are associated with the first application for which backups of files are to be stored within their own objects, and a backup of each of the combinations of files from the second set of files, associated with the second application, together as a shared object on the object storage.
2. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
identify, amongst the second set of files, a third set of files that are each larger than or equal to the predetermined object size, and a fourth set of files that are each smaller than the predetermined object size, wherein
the one or more combination of files from at least some of the second set of files are identified from the fourth set of files, and
performing the backup further includes storing the backup of each file of the third set of files as an individual object.
3. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
determine the backup of at least some of the files from the second set of files associated with the second application can be stored within shared objects.
4. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
determine the first set of files are associated with a first attribute from information retrieved from the client device, wherein
the first set of files are identified based on the determination the first set of files are associated with the first attribute.
5. The system of claim 4 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
determine the second set of files are associated with a second attribute from the information retrieved from the client device, wherein
the second set of files are identified based on the determination the second set of files are associated with the second attribute.
6. The system of claim 5 , wherein the information retrieved from the client device includes at least one of a file location, a file name, and a file type.
7. The system of claim 6 , wherein determining the backup of each file of the first set of files associated with the first application are to be stored within their own objects includes:
retrieving a backup configuration setting indicating backup files associated with the first application are to be stored within their own objects.
8. The system of claim 1 , wherein the predetermined object size is approximately 4 MB.
9. A method comprising:
receiving a request to back up files stored on a client device to an object storage;
identifying, amongst the files stored on the client device, a first set of files that are each associated with a first application, and a second set of files that are each associated with a second application;
determining a backup of each file of the first set of files associated with the first application are to be stored within their own objects;
identifying one or more combination of files from at least some of the second set of files that when grouped together are within a range of a predetermined object size; and
performing a backup of the files stored on the client device by storing the backup of each file of the first set of files as an individual object on the object storage, in response to determining that the first set of files are associated with the first application for which backups of files are to be stored within their own objects, and a backup of each of the combinations of files from the second set of files, associated with the second application, together as a shared object on the object storage.
10. The method of claim 9 , further comprising:
identifying, amongst the second set of files, a third set of files that are each larger than or equal to the predetermined object size, and a fourth set of files that are each smaller than the predetermined object size, wherein
the one or more combination of files from at least some of the second set of files are identified from the fourth set of files, and
performing the backup further includes storing the backup of each file of the third set of files as an individual object.
11. The method of claim 9 , further comprising:
determining the backup of at least some of the files from the second set of files associated with the second application can be stored within shared objects.
12. The method of claim 9 , further comprising:
determining the first set of files are associated with a first attribute from information retrieved from the client device, wherein
the first set of files are identified based on the determination the first set of files are associated with the first attribute.
13. The method of claim 12 , further comprising:
determining the second set of files are associated with a second attribute from the information retrieved from the client device, wherein
the second set of files are identified based on the determination the second set of files are associated with the second attribute.
14. The method of claim 13 , wherein the information retrieved from the client device includes at least one of a file location, a file name, and a file type.
15. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
receive a request to back up files stored on a client device to an object storage;
identify, amongst the files stored on the client device, a first set of files that are each associated with a first application, and a second set of files that are each associated with a second application;
determine a backup of each file of the first set of files associated with the first application are to be stored within their own objects;
identify one or more combination of files from at least some of the second set of files that when grouped together are within a range of a predetermined object size; and
perform a backup of the files stored on the client device by storing the backup of each file of the first set of files as an individual object on the object storage, in response to determining that the first set of files are associated with the first application for which backups of files are to be stored within their own objects, and a backup of each of the combinations of files from the second set of files, associated with the second application, together as a shared object on the object storage.
16. The computer program product of claim 15 , wherein the program code includes further instructions to:
identify, amongst the second set of files, a third set of files that are each larger than or equal to the predetermined object size, and a fourth set of files that are each smaller than the predetermined object size, wherein
the one or more combination of files from at least some of the second set of files are identified from the fourth set of files, and
performing the backup further includes storing the backup of each file of the third set of files as an individual object.
17. The computer program product of claim 15 , wherein the program code includes further instructions to:
determine the backup of at least some of the files from the second set of files associated with the second application can be stored within shared objects.
18. The computer program product of claim 15 , wherein the program code includes further instructions to:
determine the first set of files are associated with a first attribute from information retrieved from the client device, wherein
the first set of files are identified based on the determination the first set of files are associated with the first attribute.
19. The computer program product of claim 18 , wherein the program code includes further instructions to:
determine the second set of files are associated with a second attribute from the information retrieved from the client device, wherein
the second set of files are identified based on the determination the second set of files are associated with the second attribute.
20. The computer program product of claim 19 , wherein the information retrieved from the client device includes at least one of a file location, a file name, and a file type.