Encryption key rotation framework
Techniques are described herein for performing key rotation and key replacement. In an embodiment, a request is received that specifies key names. A first set of messages is generated, where each message identifies a table that is associated with the encrypted-data locations, and stored in a queue for processing by a first plurality of worker processes. Each worker process retrieves a message from the queue and generates a second message that identifies a subset of encrypted data records from the table. Each second message is stored in a distinct queue which is assigned to a worker process of a second plurality of worker processes. Each worker process retrieves the message from the assigned queue, decrypts the subset of encrypted data records, re-encrypts the decrypted data records using a new encryption key that corresponds to a new key name, and stores the re-encrypted data records in a database.
1. A computer-implemented method for key rotation, comprising:
storing one or more encrypted data records at one or more encrypted-data locations of a plurality of encrypted-data locations;
wherein each encrypted data record of the one or more encrypted data records comprises a composite value that includes:
a key version that is unencrypted,
a payload that has been encrypted with an encryption key, and
a key name that is unencrypted;
receiving a key rotation request that specifies one or more key names of a plurality of key names, a source key version, and a target key version;
for each key name of the one or more key names:
obtaining a particular encrypted data record of the one or more encrypted data records that is associated with a key name of the one or more key names and the source key version;
generating a particular decrypted data record by decrypting the payload of the particular encrypted data record using an encryption key that corresponds to the source key version;
generating a particular re-encrypted data record by encrypting the particular decrypted data record using an encryption key that corresponds to the target key version.
2. The method of claim 1 , further comprising:
storing, in one or more data repositories, a location-to-key-name mapping that maps the plurality of key names to the plurality of encrypted-data locations.
3. The method of claim 2 , wherein obtaining the particular encrypted data record of the one or more encrypted data records that is associated with the key name of the one or more key names and the source key version, comprises:
using the location-to-key-name mapping, determining the one or more encrypted-data locations of the plurality of encrypted-data locations based on the one or more key names of the plurality of key names.
4. A computer-implemented method for key replacement, comprising:
generating a first set of messages, each message of the first set of messages identifying a table associated with one or more encrypted-data locations of a plurality of encrypted-data locations;
generating, by each worker process of a first plurality of worker processes, one or more messages of a second set of messages, each of the one or more messages of the second set of messages identifying a subset of encrypted data records included in a table identified in a message of the first set of messages;
generating, by each worker process of a second plurality of worker processes, decrypted data records by decrypting the subset of encrypted data records identified in a message of the second set of messages;
generating, by each worker process of the second plurality of worker processes, re-encrypted data records by encrypting the decrypted data records that were decrypted by the respective worker process of the second plurality of worker processes;
wherein the re-encrypted data records are generating using new encryption keys to encrypt the decrypted data records.
5. The method of claim 4 , further comprising:
storing, in one or more data repositories, a location-to-key-name mapping that maps a plurality of key names to the plurality of encrypted-data locations;
receiving a key replacement request that specifies one or more key names of the plurality of key names.
6. The method of claim 5 , further comprising:
using the location-to-key-name mapping, determining the one or more encrypted-data locations of the plurality of encrypted-data locations based on the one or more key names of the plurality of key names.
7. The method of claim 5 , further comprising:
updating the location-to-key-name mapping with new key names that correspond to the new encryption keys used to encrypt the decrypted data records.
8. The method of claim 4 , further comprising:
assigning each message of the first set of messages to a distinct worker process of the first plurality of worker processes;
determining, by each worker process of the first plurality of worker processes, an amount of encrypted data records included in the table identified in the message assigned to the respective worker process of the first plurality of worker processes;
wherein a size of the subset of encrypted data records identified in each of the second set of messages is based on the amount of encrypted data records included in the table identified in the message assigned to the respective worker process of the first plurality of worker processes.
9. The method of claim 4 , further comprising:
assigning each message of the second set of messages to a distinct worker process of the second plurality of worker processes;
wherein the decrypted data records generated by each worker process of the second plurality of worker processes are generated by decrypting the subset of encrypted data records identified in the message of the second set of messages that is assigned to the respective worker process of the second plurality of worker processes.
10. The method of claim 4 , further comprising:
storing the re-encrypted data records at the one or more encrypted-data locations.
11. The method of claim 4 , further comprising:
detecting, by a particular worker process of the second plurality of worker processes, an error while generating the decrypted data records or re-encrypted data records;
generating, by the particular worker process of the second plurality of worker processes, a retry message that identifies the error;
assigning the retry message to a distinct worker process of the second plurality of worker processes for retry.
12. The method of claim 11 , further comprising:
generating a retry count based on the retry message and storing the retry count in a database.
13. The method of claim 12 , further comprising:
in response to determining that the retry count is greater than a threshold value, generating and storing a failure message in a database for manual review.
14. One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause:
storing one or more encrypted data records at one or more encrypted-data locations of a plurality of encrypted-data locations;
wherein each encrypted data record of the one or more encrypted data records comprises a composite value that includes:
a key version that is unencrypted,
a payload that has been encrypted with an encryption key, and
a key name that is unencrypted;
receiving a key rotation request that specifies one or more key names of a plurality of key names, a source key version, and a target key version;
for each key name of the one or more key names:
obtaining a particular encrypted data record of the one or more encrypted data records that is associated with a key name of the one or more key names and the source key version;
generating a particular decrypted data record by decrypting the payload of the particular encrypted data record using an encryption key that corresponds to the source key version;
generating a particular re-encrypted data record by encrypting the particular decrypted data record using an encryption key that corresponds to the target key version.
15. The one or more non-transitory computer-readable media of claim 14 , further comprising instructions for:
storing, in one or more data repositories, a location-to-key-name mapping that maps the plurality of key names to the plurality of encrypted-data locations.
16. The one or more non-transitory computer-readable media of claim 15 , wherein obtaining the particular encrypted data record of the one or more encrypted data records that is associated with the key name of the one or more key names and the source key version, comprises:
using the location-to-key-name mapping, determining the one or more encrypted-data locations of the plurality of encrypted-data locations based on the one or more key names of the plurality of key names.
17. One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause:
generating a first set of messages, each message of the first set of messages identifying a table associated with one or more encrypted-data locations of a plurality of encrypted-data locations;
generating, by each worker process of a first plurality of worker processes, one or more messages of a second set of messages, each of the one or more messages of the second set of messages identifying a subset of encrypted data records included in a table identified in a message of the first set of messages;
generating, by each worker process of a second plurality of worker processes, decrypted data records by decrypting the subset of encrypted data records identified in a message of the second set of messages;
generating, by each worker process of the second plurality of worker processes, re-encrypted data records by encrypting the decrypted data records that were decrypted by the respective worker process of the second plurality of worker processes;
wherein the re-encrypted data records are generating using new encryption keys to encrypt the decrypted data records.
18. The one or more non-transitory computer-readable media of claim 17 , further comprising instructions for:
storing, in one or more data repositories, a location-to-key-name mapping that maps a plurality of key names to the plurality of encrypted-data locations;
receiving a key replacement request that specifies one or more key names of the plurality of key names.
19. The one or more non-transitory computer-readable media of claim 18 , further comprising instructions for:
using the location-to-key-name mapping, determining the one or more encrypted-data locations of the plurality of encrypted-data locations based on the one or more key names of the plurality of key names.
20. The one or more non-transitory computer-readable media of claim 18 , further comprising instructions for:
updating the location-to-key-name mapping with new key names that correspond to the new encryption keys used to encrypt the decrypted data records.
21. The one or more non-transitory computer-readable media of claim 17 , further comprising instructions for:
assigning each message of the first set of messages to a distinct worker process of the first plurality of worker processes;
determining, by each worker process of the first plurality of worker processes, an amount of encrypted data records included in the table identified in the message assigned to the respective worker process of the first plurality of worker processes;
wherein a size of the subset of encrypted data records identified in each of the second set of messages is based on the amount of encrypted data records included in the table identified in the message assigned to the respective worker process of the first plurality of worker processes.
22. The one or more non-transitory computer-readable media of claim 17 , further comprising instructions for:
assigning each message of the second set of messages to a distinct worker process of the second plurality of worker processes;
wherein the decrypted data records generated by each worker process of the second plurality of worker processes are generated by decrypting the subset of encrypted data records identified in the message of the second set of messages that is assigned to the respective worker process of the second plurality of worker processes.
23. The one or more non-transitory computer-readable media of claim 17 , further comprising instructions for:
storing the re-encrypted data records at the one or more encrypted-data locations.
24. The one or more non-transitory computer-readable media of claim 17 , further comprising instructions for:
detecting, by a particular worker process of the second plurality of worker processes, an error while generating the decrypted data records or re-encrypted data records;
generating, by the particular worker process of the second plurality of worker processes, a retry message that identifies the error;
assigning the retry message to a distinct worker process of the second plurality of worker processes for retry.
25. The one or more non-transitory computer-readable media of claim 24 , further comprising instructions for:
generating a retry count based on the retry message and storing the retry count in a database.
26. The one or more non-transitory computer-readable media of claim 25 , further comprising instructions for:
in response to determining that the retry count is greater than a threshold value, generating and storing a failure message in a database for manual review.