Post-processing in a cloud-based data protection service
A cloud-based data protection service is disclosed. In an embodiment, the data protection service may support backup of data sets from one or more sites associated with one or more organizations. In an embodiment, deduplication of backup data across multiple sites of an organization and/or multiple sites associated with different organizations may be supported. In an embodiment, backup data may be post-processed in the cloud to insert fingerprints corresponding to data blocks that did not change since a previous backup was performed, to scan the backup for security threats such as viruses, other malware, personally identifiable information, etc. In an embodiment, restore may be supported from the cloud, where restore blocks may be larger than backup data blocks. In another embodiment, restore may be based on blocks that have changed since the most recent backup (or a user-selected backup).
1. A non-transitory computer accessible storage medium storing a plurality of instructions forming a data protection service executable in a cloud, where the plurality of instructions, when executed on one or more computer systems, cause the one or more computer systems to perform operations comprising:
receiving, from a site, a first data descriptor corresponding to a complete data set for a backup of the site, wherein the site comprises a physical location at which a data center belonging to an organization is located, wherein the first data descriptor comprises a list of fingerprints and is partially-populated at the site such that:
the list includes fingerprints identifying data blocks in corresponding backup data from the site for data blocks that have changed from a previous backup from the site; and
the list includes unpopulated entries for excluded fingerprints for data blocks that have not changed from the previous backup from the site;
obtaining, by the data protection service, a second data descriptor corresponding to the previous backup, wherein the second data descriptor is obtained from the cloud; and
merging, by the data protection service, fingerprints from the second data descriptor into the unpopulated entries in the first data descriptor to complete the first data descriptor.
2. The non-transitory computer accessible storage medium as recited in claim 1 wherein the receiving the first data descriptor comprises:
receiving an identifier for the first data descriptor, the identifier identifying the first data descriptor in an object storage of a cloud that hosts the data protection service; and
getting the first data descriptor from the object storage.
3. The non-transitory computer accessible storage medium as recited in claim 2 wherein the operations further comprise putting the completed first data descriptor in the object storage using the identifier, overwriting the first data descriptor that is partially-populated in the object storage.
4. The non-transitory computer accessible storage medium as recited in claim 1 wherein the operations further comprise:
restoring the backup in the cloud; and
verifying the restored backup.
5. The non-transitory computer accessible storage medium as recited in claim 4 wherein the backup corresponds to a first virtual machine from the site, and wherein restoring the backup comprises establishing a second virtual machine in the cloud and restoring the backup to a block storage provided to the second virtual machine.
6. The non-transitory computer accessible storage medium as recited in claim 5 wherein establishing the second virtual machine comprises successfully bidding for a spot instance of the second virtual machine.
7. The non-transitory computer accessible storage medium as recited in claim 5 wherein establishing the second virtual machine comprises unsuccessfully bidding for a spot instance of the second virtual machine and establishing an on-demand instance of the second virtual machine.
8. The non-transitory computer accessible storage medium as recited in claim 4 wherein the operations further comprise generating a file listing from the restored backup.
9. The non-transitory computer accessible storage medium as recited in claim 8 wherein the operations further comprise putting the file listing in an object storage.
10. The non-transitory computer accessible storage medium as recited in claim 4 wherein the operations further comprise:
scanning the restored backup for one or more security risks.
11. The non-transitory computer accessible storage medium as recited in claim 10 wherein scanning the restored backup comprises scanning the backup for viruses.
12. The non-transitory computer accessible storage medium as recited in claim 10 wherein scanning the restored backup comprises scanning the backup for malware.
13. The non-transitory computer accessible storage medium as recited in claim 10 wherein scanning the restored backup comprises scanning the backup for personally identifiable information.
14. The non-transitory computer accessible storage medium as recited in claim 1
wherein a particular one of the unpopulated entries includes a fingerprint of all zero bytes; and
wherein the merging includes replacing the all zero bytes with a valid fingerprint from the second descriptor.
15. A method comprising:
receiving, by a data protection service executable in a cloud and from a site, a first data descriptor corresponding to a backup of the site, wherein the site comprises a physical location at which a data center belonging to an organization is located, wherein the first data descriptor comprises a list of fingerprints for a complete backup data set, and wherein the first data descriptor is partially-populated at the site such that:
the list includes fingerprints identifying data blocks in corresponding backup data from the site for data blocks that have changed from a previous backup from the site; and
the list includes unpopulated locations for excluded fingerprints for data blocks that have not changed from the previous backup from the site;
retrieving, by the data protection service, a second data descriptor corresponding to the previous backup, wherein the second data descriptor is retrieved from the cloud; and
merging, by the data protection service, fingerprints from the second data descriptor into the unpopulated locations in the first data descriptor to complete the first data descriptor.
16. The method as recited in claim 15 further comprising putting the completed first data descriptor in an object storage, overwriting the first data descriptor that is partially-populated in the object storage.
17. The method as recited in claim 15 further comprising:
restoring the backup in a cloud; and
verifying the restored backup.
18. The method as recited in claim 17 further comprising:
scanning the restored backup for one or more security risks.
19. The method as recited in claim 17 further comprising:
scanning the restored backup for personally identifiable information.
20. One or more computer systems, each of the one or more computer systems comprising one or more processors, the one or more processors coupled to a non-transitory computer accessible storage medium storing a plurality of instructions forming a data protection service executable in a cloud which, when executed by the one or more processors, cause the one or more computer systems to perform operations comprising:
receiving, from a site, a first data descriptor corresponding to a backup from the site, wherein the site comprises a physical location at which a data center belonging to an organization is located, wherein the first data descriptor comprises a list of fingerprints having locations for a complete backup data set, and wherein the first data descriptor is partially-populated at the site such that:
the list includes populated locations with fingerprints identifying data blocks in corresponding backup data from the site for data blocks that have changed from a previous backup from the site; and
the list includes unpopulated locations excluding fingerprints for data blocks that have not changed from the previous backup from the site;
obtaining, by the data protection service, a second data descriptor corresponding to the previous backup, wherein the second data descriptor is obtained from the cloud; and
merging, by the data protection service, fingerprints from the second data descriptor into the unpopulated locations in the first data descriptor to complete the first data descriptor, including replacing a placeholder value the fingerprints from the second descriptor.