Snapshot management across cloud provider network extension security boundaries
Systems and methods for efficient and secure management of encrypted “snapshots” for a remote provider substrate extension (“PSE”) of a cloud provider network substrate are provided. The PSE may request and obtain a snapshot from the cloud provider network substrate, restore a volume from the snapshot, make changes to data in the restored volume, and/or initiate the creation and storage of a new snapshot that includes incremental updates to the original snapshot to reflect the changes made to data in the volume. An encrypted snapshot stored within the cloud provider network substrate may be decrypted using a cloud provider key designed for internal use only, and then re-encrypted using a PSE-specific key before providing the snapshot to the PSE, thereby avoiding the sharing of the cloud provider internal use only key outside the cloud provider network substrate.
1. A cloud provider system comprising:
an internal object data store;
an externally-accessible object data store; and
one or more computing devices configured to at least:
store, in the internal object data store, an object storage representation of a data volume,
wherein the object storage representation comprises a first plurality of data objects encrypted using a first extension-specific key,
wherein the first extension-specific key is permitted to be accessed by a first extension of the cloud provider system,
wherein the first extension comprises a first set of computing resources located at a first customer site remote from a region of the cloud provider system and managed by the cloud provider system,
and wherein the first set of computing resources provides, at the first customer site, on-premise computing services corresponding to at least a portion of cloud-based computing services provided by a second set of computing resources within the cloud provider system;
receive a request to restore a block storage representation of the data volume to a second extension of the cloud provider system using the object storage representation, wherein the second extension comprises a third set of computing resources located at a second customer site remote from the region of the cloud provider system and managed by the cloud provider system;
determine that the second extension of the cloud provider system is prohibited from accessing the first extension-specific key;
generate an export version of the object storage representation, wherein the export version comprises a second plurality of data objects encrypted using a second extension-specific key permitted to be accessed by the second extension of the cloud provider system; and
store the export version in the externally-accessible object data store, wherein the second extension of the cloud provider system accesses the export version from the externally-accessible object data store.
2. The cloud provider system of claim 1 , wherein the one or more computing devices are further configured to:
receive a second request to restore a second block storage representation of the data volume to a block store server of the cloud provider system;
determine that the block store server is permitted to access data encrypted with the first extension-specific key; and
restore the second block storage representation to the block store server using the first extension-specific key.
3. The cloud provider system of claim 1 , wherein the one or more computing devices are further configured to:
receive, in the externally-accessible object data store, a third plurality of data objects from the second extension of the cloud provider system, wherein individual objects of the third plurality of data objects comprise modified versions of individual objects of the second plurality of data objects, and wherein the third plurality of data objects comprises fewer objects than the second plurality of data objects;
generate an incremental update version of the object storage representation comprising the third plurality of data objects and a subset of the second plurality of data objects, wherein the incremental update version is encrypted based on the second extension-specific key, and wherein the third plurality of data objects does not include modified versions of individual objects of the subset of the second plurality of data objects; and
store the incremental update version in the internal object data store.
4. A computer-implemented method comprising:
under control of a provider network comprising one or more computing devices configured to execute specific instructions,
receiving a request to restore a snapshot of a data volume to a destination extension of the provider network, wherein the destination extension comprises a first set of computing resources located at a customer site remote from a region of the provider network and managed by the provider network, and wherein a first version of the snapshot is stored in an internal data store of the provider network;
determining that the first version of the snapshot is encrypted using a first key permitted to be accessed by a source extension of the provider network, wherein the source extension comprises a second set of computing resources different from the first set of computing resources;
determining that the destination extension is prohibited from accessing the first key; and
generating a second version of the snapshot based at least partly on the first version of the snapshot, wherein the second version of the snapshot is encrypted using a second key permitted to be accessed by the destination extension.
5. The computer-implemented method of claim 4 , wherein determining that the destination extension is prohibited from accessing the first key is based at least partly on: the first key being associated with the source extension; and the source extension being different than the destination extension.
6. The computer-implemented method of claim 4 , further comprising storing the second version of the snapshot in an externally-accessible data store, wherein the destination extension obtains the second version of the snapshot from the externally-accessible data store.
7. The computer-implemented method of claim 4 , further comprising:
generating a first export list referencing a first subset of a plurality of portions of the first version of the snapshot;
generating a second export list referencing a second subset of the plurality of portions of the first version of the snapshot;
re-encrypting the first subset of the plurality of portions using the first export list, the first key, and the second key; and
re-encrypting the second subset of the plurality of portions using the second export list, the first key, and the second key.
8. The computer-implemented method of claim 7 , wherein re-encrypting the first subset of the plurality of portions occurs substantially in parallel with re-encrypting the second subset of the plurality of portions.
9. The computer-implemented method of claim 7 , wherein re-encrypting the first subset of the plurality of portions comprises:
decrypting the first subset of the plurality of portions using the first key to generate a decrypted subset of the plurality of portions; and
encrypting the decrypted subset of the plurality of portions using the second key.
10. The computer-implemented method of claim 4 , further comprising:
receiving a second request to restore the snapshot to a server of the provider network;
determining that the server is permitted to access data encrypted with the first key; and
restoring the snapshot to the server using the first key.
11. The computer-implemented method of claim 4 , further comprising:
receiving, in an externally-accessible data store, a change set portion from the destination extension, wherein the change set portion comprises a modified version of a portion of the snapshot, and wherein the change set portion is encrypted using the second key;
generating a second snapshot using the change set portion and the second version of the snapshot; and
storing the second snapshot in the internal data store.
12. A system comprising:
a substrate network;
an internal data store; and
one or more computing devices in communication with the internal data store via the substrate network, wherein the one or more computing devices are configured to at least:
receive a request to restore a data snapshot to a destination network extension comprising a first set of computing resources located at a first customer site remote from a region of the substrate network and managed by the substrate network, wherein a first version of the data snapshot is stored in the internal data store;
determine that the first version of the data snapshot is encrypted using a first key permitted to be accessed by a source network extension comprising a second set of computing resources located at a second customer site remote from the region of the substrate network and managed by the substrate network;
determine that the destination network extension is prohibited from accessing data encrypted using the first key; and
generate a second version of the data snapshot based at least partly on the first version of the data snapshot, wherein the second version of the data snapshot is encrypted using a second key permitted to be accessed by the destination network extension.
13. The system of claim 12 , wherein the destination network extension is determined to be prohibited from accessing data encrypted using first key based at least partly on: the first key being associated with the source network extension; and the source network extension being different than the destination network extension.
14. The system of claim 12 , wherein the one or more computing devices are further configured to store the second version of the data snapshot in an externally-accessible data store, wherein the destination network extension obtains the second version of the data snapshot from the externally-accessible data store.
15. The system of claim 12 , wherein the one or more computing devices are further configured to at least:
generate a first export list referencing a first subset of a plurality of portions of the first version of the data snapshot;
generate a second export list referencing a second subset of the plurality of portions of the first version of the data snapshot;
re-encrypt the first subset of the plurality of portions using the first export list, the first key, and the second key; and
re-encrypt the second subset of the plurality of portions using the second export list, the first key, and the second key, wherein re-encryption of the first subset of the plurality of portions occurs substantially in parallel with re-encryption of the second subset of the plurality of portions.
16. The system of claim 15 , wherein the one or more computing devices configured to re-encrypt the first subset of the plurality of portions are further configured to at least:
decrypt the first subset of the plurality of portions using the first key to generate a decrypted subset of the plurality of portions; and
encrypt the decrypted subset of the plurality of portions using the second key.
17. The system of claim 12 , wherein the data snapshot comprises one of: a data volume backup, or a machine image.
18. The system of claim 12 , wherein the one or more computing devices are further configured to at least:
receive a second request to restore the data snapshot to a server of the substrate network;
determine that the server is permitted to access data encrypted with the first key; and
restore the data snapshot to the server using the first key.
19. The system of claim 12 , wherein the one or more computing devices are further configured to at least:
receive, in an externally-accessible data store, a change set portion from the destination network extension, wherein the change set portion comprises a modified version of a portion of the data snapshot, and wherein the change set portion is encrypted using the second key;
generate a second data snapshot using the change set portion and the second version of the data snapshot; and
store the second data snapshot in the internal data store.
20. A computer-implemented method comprising:
maintaining key data in a region of a cloud provider network, the key data associating a first key with a first extension of the cloud provider network and associating a second key with a second extension of the cloud provider network,
wherein the first extension is located at a first customer site remote from a region of the cloud provider network and managed by the cloud provider network,
wherein the first extension comprises a first set of computing resources different from a second set of computing resources of the second extension,
wherein the second extension is located at a second customer site remote from the region of the cloud provider network and managed by the cloud provider network,
wherein usage of the first key is limited to the first extension and to an encryption service running within the region of the cloud provider network,
wherein usage of the second key is limited to the second extension and to the encryption service, and
wherein the first key is different from the second key;
receiving a request to use a snapshot of a data volume in the second extension, wherein the snapshot was generated at least partly in the first extension of the cloud provider network;
determining that a first version of the snapshot is stored in a data store within the region of the cloud provider network and is encrypted using the first key limited to use by the first extension and the encryption service;
using the encryption service to generate a second version of the snapshot based at least partly on re-encrypting the first version of the snapshot using the second key limited to use by the second extension and the encryption service; and
transmitting the second version of the snapshot to the second extension.
21. The computer-implemented method of claim 20 , further comprising:
maintaining a first manifest for the first version of the snapshot; and
maintaining a second manifest for the second version of the snapshot;
wherein the first and second manifests comprise substantially identical storage path information for corresponding portions of the snapshot.
22. The cloud provider system of claim 1 , further comprising a plurality of availability zones,
wherein a first availability zone of the plurality of availability zones comprises a first subset of cloud provider resources,
wherein a second availability zone of the plurality of availability zones comprises a second subset of cloud provider resources,
wherein the first availability zone is in communication with the second availability zone over a private network, and
wherein the first availability zone is in communication with the first extension of the cloud provider system over a public network.