Enhancing bulk export of backup data
Provided are mechanisms and processes for a improving the processing and network efficiency of cloud backup bulk data export operations. Data being restored from cloud backups and archives to multiple devices can be intelligently analyzed in order to reduce consumption of network and processing resources on wide area networks and at cloud backup service providers. Data analysis may involve generating blockmaps, analyzing metadata, and determining other data attributes in order to reduce redundancy, determine tasks that can be handled locally, and benefit from other data characteristics.
1 . A method, comprising:
receiving at a server a request to bulk export data to a plurality of users including a first user and a second user, the bulk export data comprising a plurality of files to be restored onto a first user device associated with the first user and a second user device associated with the second user, wherein the server is connected to the first and second user devices over a wide area network and the first and second user devices are connected to each other over a local area network;
traversing a first plurality of files to be restored for a first user;
restoring the first plurality of files to the first user by transmitting the first plurality of files over the wide area network;
identifying a first subset of files and a second subset of files in the first plurality of files, the first subset of files common to the first and the second user and a second subset of files not common to the first and the second user;
restoring the first subset of files at a file level to the second user by copying the first subset of files from the first user to the second user over a local area network;
restoring the second subset of files at a block level to the second user by determining common elements of the second subset of files to be restored for the second user, wherein common elements are determined by referencing a plurality of blockmaps associated with the first plurality of files and the second plurality of files, wherein common elements are further determined by traversing a reverse map that, for each block handle, identifies a user identifier, a filepath, and an offset within the filepath that references the block;
restoring the common elements of the second subset of files the second user by transmitting the common elements over a local area network from the first user device to the second user device; and
restoring a second plurality of files to a second user by transmitting the second plurality of files minus the common elements over a wide area network to the second user.
2 . The method of claim 1 , wherein the common elements comprise a first plurality of identical files to be restored to both the first user and the second user.
3 . The method of claim 2 , wherein the first plurality of identical files are determined to be identical based on identical checksums.
4 . The method of claim 2 , wherein the first plurality of identical files are deduplicated files.
5 . The method of claim 1 , wherein the common elements comprise a first plurality of identical blocks to be restored to both the first user and the second user, wherein each of the first plurality of files includes a plurality of blocks.
6 . The method of claim 5 , wherein the first plurality of identical blocks are determined using a plurality of blockmaps, each of the first plurality of files including a corresponding blockmap.
7 . The method of claim 5 , wherein a first file in the first plurality of files comprises a first blockmap.
8 . The method of claim 7 , wherein the first blockmap comprises a plurality of objects identifiers corresponding to a plurality of offsets.
9 . The method of claim 8 , wherein the first blockmap further comprises attributes including shareinfo indicating whether a first block is shared with other user devices.
10 . The method of claim 9 , wherein the shareinfo indicates that the first object is shared with the first user, the second user, and a third user.
11 . A non-transitory computer readable medium, comprising:
computer code for receiving at a server a request to bulk export data to a plurality of users including a first user and a second user, the bulk export data comprising a plurality of files to be restored onto a first user device associated with the first user and a second user device associated with the second user, wherein the server is connected to the first and second user devices over a wide area network and the first and second user devices are connected to each other over a local area network;
computer code for traversing a first plurality of files to be restored for a first user;
computer code for restoring the first plurality of files to the first user by transmitting the first plurality of files over the wide area network;
computer code for identifying a first subset of files and a second subset of files in the first plurality of files, the first subset of files common to the first and the second user and a second subset of files not common to the first and the second user;
computer code for restoring the first subset of files at a file level to the second user by copying the first subset of files from the first user to the second user over a local area network;
computer code for restoring the second subset of files at a block level to the second user by determining common elements of the second subset of files to be restored for the second user, wherein common elements are determined by referencing a plurality of blockmaps associated with the first plurality of files and the second plurality of files, wherein common elements are further determined by traversing a reverse map that, for each block handle, identifies a user identifier, a filepath, and an offset within the filepath that references the block;
computer code for restoring the common elements of the second subset of files to the second user by transmitting the common elements over a local area network from the first user device to the second user device; and
computer code for restoring a second plurality of files to a second user by transmitting the second plurality of files minus the common elements over a wide area network to the second user.
12 . The computer code of claim 11 , wherein the common elements comprise a first plurality of identical files to be restored to both the first user and the second user.
13 . The computer code of claim 12 , wherein the first plurality of identical files are determined to be identical based on identical checksums.
14 . The computer code of claim 12 , wherein the first plurality of identical files are deduplicated files.
15 . The computer code of claim 11 , wherein the common elements comprise a first plurality of identical blocks to be restored to both the first user and the second user, wherein each of the first plurality of files includes a plurality of blocks.
16 . The computer code of claim 15 , wherein the first plurality of identical blocks are determined using a plurality of blockmaps, each of the first plurality of files including a corresponding blockmap.
17 . The computer code of claim 15 , wherein a first file in the first plurality of files comprises a first blockmap.
18 . The computer code of claim 17 , wherein the first blockmap comprises a plurality of objects identifiers corresponding to a plurality of offsets.
19 . The computer code of claim 18 , wherein the first blockmap further comprises attributes including shareinfo indicating whether a first block is shared with other user devices.
20 . The computer code of claim 19 , wherein the shareinfo indicates that the first object is shared with the first user, the second user, and a third user.