Data backup and progressive restoration using data chunks in cloud storage and a data cache
A system for progressive just-in-time restoration of data from backup media. Backup data may be divided into a plurality of chunks and stored on any kind of media such as a direct attached storage (DAS) disk, object storage, USB drive, network share or tape. An index map is maintained that indicates the location of each of the plurality of chunks in cloud storage, the index map representing contiguous blocks of backup data of a volume. The backup data may be compressed, encrypted, or de-duplicated. The backup data may be located on different media, object stores, or network shares, or at differing geographic locations. To perform a recovery, a virtual LUN or virtual volume is mounted and provided to the operating system and applications of the restored computer. Chunks may be progressively copied from cloud storage to a data cache and restored in response to requests for blocks.
1. A method for providing a mountable backup, comprising:
creating a base backup of a source volume;
dividing the base backup into a plurality of chunks, each chunk comprising a plurality of blocks;
copying each of the plurality of chunks to a location in a cloud storage;
maintaining in the cloud storage an index indicating the location of each of the plurality of chunks in the cloud storage, a copy of the index being stored with each of the plurality of chunks;
creating a plurality of subsequent delta backups, the plurality of subsequent delta backups each referring to at least one of the plurality of chunks;
providing a virtual volume comprising a data cache and a cloud storage interface, the virtual volume operative to provide restoration at a selected time by merging the base backup and the plurality of subsequent delta backups;
mounting the virtual volume on a computer system;
receiving at the virtual volume a read request for a first block;
determining from the index a first location in the cloud storage associated with a first chunk of the plurality of chunks, the first chunk comprising the first block;
copying the first chunk from the first location in the cloud storage to the data cache via the cloud storage interface;
fulfilling the read request from the data cache;
receiving at the virtual volume a write request for write data;
caching the write data, into the first chunk in the data cache; and
after a set interval, sending the first chunk containing the write data from the data cache to the cloud storage.
2. The method of claim 1 , wherein at least one of the plurality of chunks is copied concurrently with at least another of the plurality of chunks.
3. The method of claim 1 , further comprising:
copying each of the plurality of chunks to an additional location in the cloud storage.
4. The method of claim 1 , further comprising:
selecting the location in the cloud storage based on its locality to the source volume.
5. The method of claim 4 , wherein locality is measured by network latency.
6. The method of claim 4 , wherein locality is measured by network bandwidth.
7. The method of claim 4 , wherein locality is measured by physical proximity.
8. The method of claim 1 , further comprising:
removing duplicate chunks from the plurality of chunks.
9. The method of claim 1 , further comprising:
maintaining in a central database an additional copy of the index.