Streaming slices out of order for efficient backup
Embodiments of the invention relate to methods and systems for efficiently backing up and restoring assets such as files and folders in the form of slices and streams. In one or more embodiments, assets such as files and folders that are to be backed up are placed in appropriate network slice queues, which are then assigned to network streams. The slices/assets are assigned in such a way that each stream has a similar cost and throughput. The backup is performed, and the streams are stored in their own backup containers until a restoration is needed. Once a restoration is needed, the meta-data for the backup is used to restore the files and folders, forming the asset back to their appropriate locations in the target file-system. This results in efficient use of network resources, while minimizing the amount of work the hosts and/or clients must perform during a backup or restoration.
1. A method for performing a backup, the method comprising:
receiving a request for a backup of an asset associated with a production host, wherein the asset comprises files and folders stored in a file-system of the production host;
in response to the request, retrieving network statistics for a plurality of streams;
determining, based on the network statistics, peak throughput for the plurality of streams;
retrieving meta-data for the asset;
slicing, using the meta-data for the asset, the asset into a plurality of slices;
assigning each of the plurality of slices to one of the plurality of streams in such a way that each stream of the plurality of streams has a similar throughput based on the peak throughput for each of the plurality of streams;
after the assigning:
initiating the backup using the plurality of streams;
storing data transferred in each stream of the plurality of streams to its own backup container, wherein the data comprises files and folder in each slices of the asset assigned to each stream of the plurality of streams;
generating mapping, wherein the mapping maps each file and folder in each slice of the asset stored in the backup container to its original location in the file-system of the production host; and
storing the mapping in meta-data associated with the backup.
2. The method of claim 1 , the method further comprising:
after storing the mapping, receiving a request to restore the asset;
retrieving, in response to receiving the request, the meta-data associated with the backup;
determining, using the mapping, which container includes each file and folder forming the asset; and
restoring, using the mapping, each file and folder forming the asset to its original location in the file-system of a target production host.
3. The method of claim 2 , wherein during restoring, the slices are not moved to a different backup container.
4. The method of claim 1 , wherein the files and folders comprising the asset are not backed up in order.
5. The method of claim 1 , wherein when the slices are assigned to one of the plurality of streams in such a way that each stream of the plurality of streams has the similar throughput, the slices are assigned such that each stream of the plurality of streams has a total throughput that is within a predetermined percentage of the total throughput of each of the other streams of the plurality of streams.
6. The method of claim 5 , wherein each stream of the plurality of streams also has a similar total cost that is calculated for each slice, wherein the cost is calculated based on the number of times that each file and folder that comprises each slice must be open, read, and closed when the backup is performed.
7. The method of claim 6 , wherein each stream of the plurality of streams has a total cost that is within a predetermined percentage of the total cost of each of the other streams of the plurality of streams.
8. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor, enables the computer processor to perform a method for performing a backup, the method comprising:
receiving a request for a backup of an asset associated with a production host, wherein the asset comprises files and folders stored in a file-system of the production host;
in response to the request, retrieving network statistics for a plurality of streams;
determining, based on the network statistics, peak throughput for the plurality of streams;
retrieving meta-data for the asset;
slicing, using the meta-data for the asset, the asset into a plurality of slices;
assigning each of the plurality of slices to one of the plurality of streams in such a way that each stream of the plurality of streams has a similar throughput based on the peak throughput for each of the plurality of streams;
after the assigning:
initiating the backup using the plurality of streams;
storing data transferred in each stream of the plurality of streams to its own backup container, wherein the data comprises files and folder in each slices of the asset assigned to each stream of the plurality of streams;
generating mapping, wherein the mapping maps each file and folder in each slice of the asset stored in the backup container to its original location in the file-system of the production host; and
storing the mapping in meta-data associated with the backup.
9. The non-transitory computer readable medium of claim 8 , wherein the method further comprises:
after storing the mapping, receiving a request to restore the asset;
retrieving, in response to receiving the request, the meta-data associated with the backup;
determining, using the mapping, which container includes each file and folder forming the asset; and
restoring, using the mapping, each file and folder forming the asset to its original location in the file-system of a target production host.
10. The non-transitory computer readable medium of claim 9 , wherein during restoring, the slices are not moved to a different backup container.
11. The non-transitory computer readable medium of claim 8 , wherein the files and folders comprising the asset are not backed up in order.
12. The non-transitory computer readable medium of claim 8 , wherein when the slices are assigned to one of the plurality of streams in such a way that each stream of the plurality of streams has the similar throughput, the slices are assigned such that each stream of the plurality of streams has a total throughput that is within a predetermined percentage of the total throughput of each of the other streams of the plurality of streams.
13. The non-transitory computer readable medium of claim 12 , wherein each stream of the plurality of streams also has a similar total cost that is calculated for each slice, wherein the cost is calculated based on the number of times that each file and folder that comprises each slice must be open, read, and closed when the backup is performed.
14. The non-transitory computer readable medium of claim 13 , wherein each stream of the plurality of streams has a total cost that is within a predetermined percentage of the total cost of each of the other streams of the plurality of streams.
15. A system comprising:
a production host which comprises of:
a processor; and
a memory comprising instructions, which when executed by the processor, perform a method for performing a backup comprising:
receiving a request for a backup of an asset associated with a production host, wherein the asset comprises files and folders stored in a file-system of the production host;
in response to the request, retrieving network statistics for a plurality of streams;
determining, based on the network statistics, peak throughput for the plurality of streams;
retrieving meta-data for the asset;
slicing, using the meta-data for the asset, the asset into a plurality of slices;
assigning each of the plurality of slices to one of the plurality of streams in such a way that each stream of the plurality of streams has a similar throughput based on the peak throughput for each of the plurality of streams;
after the assigning:
initiating the backup using the plurality of streams;
storing data transferred in each stream of the plurality of streams to its own backup container, wherein the data comprises files and folder in each slices of the asset assigned to each stream of the plurality of streams;
generating mapping, wherein the mapping maps each file and folder in each slice of the asset stored in the backup container to its original location in the file-system of the production host; and
storing the mapping in meta-data associated with the backup.
16. The system of claim 15 , wherein the method further comprises:
after storing the mapping, receiving a request to restore the asset;
retrieving, in response to receiving the request, the meta-data associated with the backup;
determining, using the mapping, which container includes each file and folder forming the asset; and
restoring, using the mapping, each file and folder forming the asset to its original location in the file-system of a target production host.
17. The system of claim 16 , wherein during restoring, the slices are not moved to a different backup container.
18. The system of claim 15 , wherein the files and folders comprising the asset are not backed up in order.
19. The system of claim 15 , wherein when the slices are assigned to one of the plurality of streams in such a way that each stream of the plurality of streams has the similar throughput, the slices are assigned such that each stream of the plurality of streams has a total throughput that is within a predetermined percentage of the total throughput of each of the other streams of the plurality of streams.
20. The system of claim 19 , wherein each stream of the plurality of streams also has a similar total cost that is calculated for each slice, wherein the cost is calculated based on the number of times that each file and folder that comprises each slice must be open, read, and closed when the backup is performed.