Provisioning of a shippable storage device and ingesting data from the shippable storage device
When a client requests a data import job, a remote storage service provider provisions a shippable storage device that will be used to transfer client data from the client to the service provider for import. The service provider generates security information for the data import job, provisions the shippable storage device with the security information, and sends the shippable storage device to the client. The service provider also sends client-keys to the client, separate from the shippable storage device (e.g., via a network). The client receives the device, encrypts the client data and keys, transfers the encrypted data and keys onto the device, and ships it back to the service provider. The remote storage service provider authenticates the storage device, decrypts client-generated keys using the client-keys stored at the storage service provider, decrypts the data using the decrypted client-side generated keys, and imports the decrypted data.
1. A system, comprising:
one or more processors; and
one or more memories, wherein the one or more memories have stored thereon instructions, which when executed by the one or more processors, cause the one or more processors to implement a data ingestion service, wherein the data ingestion service is configured to:
determine a data import job associated with a shippable storage device received by the storage service provider from a client;
obtain, based on the data import job, one or more stored keys stored by the storage service provider;
obtain encrypted keys associated with the data import job;
obtain encrypted data from the shippable storage device;
decrypt one or more of the encrypted keys using the one or more stored keys to generate one or more decrypted keys;
decrypt the encrypted data based on usage of the one or more decrypted keys to generate decrypted data; and
store the decrypted data at one or more locations at the storage service provider.
2. The system as recited in claim 1 , wherein the encrypted keys comprise one or more encrypted chunk keys, the one or more decrypted keys comprise one or more file keys, the encrypted data comprises one or more encrypted chunks of data, the decrypted data comprises one or more decrypted chunks of data, and wherein to decrypt the encrypted data based on usage of the one or more decrypted keys, the data ingestion service is further configured to:
decrypt the one or more encrypted chunk keys using the one or more file keys to generate one or more chunk keys; and
decrypt the one or more encrypted chunks of data using the one or more chunk keys to generate the decrypted data.
3. The system as recited in claim 1 , wherein the data ingestion service is further configured to:
receive configuration information from a trusted platform module of the shippable storage device; and
determine, based on the configuration information, that a configuration of the shippable storage device has not changed since the shippable storage device was provisioned and sent to the client.
4. The system as recited in claim 1 , wherein the data ingestion service is further configured to:
obtain, based on the information for the data import job, security information and one or more stored keys stored by the storage service provider; and
authenticate the shippable storage device based on the security information.
5. The system as recited in claim 1 , wherein to obtain encrypted keys associated with the data import job, the data ingestion service is configured to:
obtain, based on the data import job, the encrypted keys from a storage location of the storage service provider, or
obtain the encrypted keys from the shippable storage device.
6. The system as recited in claim 1 , wherein the decrypted data comprises a shard, and wherein the data ingestion service is configured to:
reconstruct data based on the shard and at least one other shard; and
store the reconstructed data at the one or more locations at the storage service provider.
7. A method, comprising:
performing, by a data ingestion service implemented on one or more computing devices of a storage service provider:
determining a data import job associated with a shippable storage device received by the storage service provider from a client;
obtaining, based on the data import job, one or more stored keys stored by the storage service provider;
obtaining encrypted keys associated with the data import job;
obtaining encrypted data from the shippable storage device;
decrypting one or more of the encrypted keys using the one or more stored keys to generate one or more decrypted keys;
decrypting the encrypted data based on usage of the one or more decrypted keys to generate decrypted data; and
storing the decrypted data at one or more locations at the storage service provider.
8. The method as recited in claim 7 , wherein the encrypted keys comprise one or more encrypted chunk keys, the one or more decrypted keys comprise one or more file keys, the encrypted data comprises one or more encrypted chunks of data, the decrypted data comprises one or more decrypted chunks of data, and wherein decrypting the encrypted data based on usage of the one or more decrypted keys comprises:
decrypting the one or more encrypted chunk keys using the one or more file keys to generate one or more chunk keys; and
decrypting the one or more encrypted chunks of data using the one or more chunk keys to generate the one or more decrypted chunks of data.
9. The method as recited in claim 8 , further comprising:
combining two or more of the decrypted chunks of data to form a file.
10. The method as recited in claim 9 , further comprising:
calculating a checksum for the file;
comparing the calculated checksum to a checksum stored by the storage service provider; and
verifying the integrity of the file in response to determining that the calculated checksum matches the stored checksum.
11. The method as recited in claim 7 , wherein the decrypted data comprises a shard, and further comprising:
reconstructing data based on the shard and at least one other shard; and
storing the reconstructed data at the one or more locations at the storage service provider.
12. The method as recited in claim 7 , further comprising:
receiving configuration information from a trusted platform module of the shippable storage device; and
determining, based on the configuration information, that a configuration of the shippable storage device has not changed since the shippable storage device was provisioned and sent to the client.
13. The method as recited in claim 7 , wherein obtaining encrypted keys associated with the data import job comprises:
obtaining, based on the data import job, the encrypted keys from a storage location of the storage service provider, or
obtaining the encrypted keys from the shippable storage device.
14. One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors cause the one or more processors to implement an event-driven compute service to:
determine a data import job associated with a shippable storage device received by the storage service provider from a client;
obtain, based on the data import job, one or more stored keys stored by the storage service provider;
obtain encrypted keys associated with the data import job;
obtain encrypted data from the shippable storage device;
decrypt one or more of the encrypted keys using the one or more stored keys to generate one or more decrypted keys;
decrypt the encrypted data based on usage of the one or more decrypted keys to generate decrypted data; and
store the decrypted data at one or more locations at the storage service provider.
15. The one or more storage media as recited in claim 14 , wherein the encrypted keys comprise one or more encrypted chunk keys, the one or more decrypted keys comprise one or more file keys, the encrypted data comprises one or more encrypted chunks of data, the decrypted data comprises one or more decrypted chunks of data, and wherein to decrypt the encrypted data based on usage of the one or more decrypted keys, the program instructions when executed on or across the one or more processors cause the one or more processors to:
decrypt the one or more encrypted chunk keys using the one or more file keys to generate one or more chunk keys; and
decrypt the one or more encrypted chunks of data using the one or more chunk keys to generate the one or more decrypted chunks of data.
16. The one or more storage media as recited in claim 15 , further comprising program instructions that when executed on or across the one or more processors cause the one or more processors to:
combine two or more of the decrypted chunks of data to form a file.
17. The one or more storage media as recited in claim 16 , further comprising program instructions that when executed on or across the one or more processors cause the one or more processors to:
calculate a checksum for the file;
compare the calculated checksum to a checksum stored by the storage service provider; and
verify the integrity of the file in response to determining that the calculated checksum matches the stored checksum.
18. The one or more storage media as recited in claim 14 , wherein the decrypted data comprises a shard, and further comprising program instructions that when executed on or across the one or more processors cause the one or more processors to:
reconstruct data based on the shard and at least one other shard; and
store the reconstructed data at the one or more locations at the storage service provider.
19. The one or more storage media as recited in claim 14 , further comprising program instructions that when executed on or across the one or more processors cause the one or more processors to:
receive configuration information from a trusted platform module of the shippable storage device; and
determine, based on the configuration information, that a configuration of the shippable storage device has not changed since the shippable storage device was provisioned and sent to the client.
20. The one or more storage media as recited in claim 14 , further comprising program instructions that when executed on or across the one or more processors cause the one or more processors to:
obtain, based on the data import job, a root certificate or another key stored by the storage service provider; and
authenticate the shippable storage device based at least on the root certificate or the other key.