On-demand encryption for online database encryption transitions
On-demand encryption may be implemented for online database encryption transition. Determinations can be made for individual data blocks as to whether individual data blocks have been encrypted using a new encryption scheme. Therefore read requests, write requests, or background operations can access and encrypt, if not already performed, different data blocks for the database while the database is in transition to the new encryption scheme.
1 . A system, comprising:
at least one processor; and
a memory, storing program instructions that when executed by the at least one processor, cause the at least one processor to implement a database system, configured to:
receive a query that causes one or more data blocks of a plurality of data blocks in a storage system storing a database to be read, wherein the database is in a transition state to a new encryption scheme;
evaluate metadata comprising individual encryption transition state maintained as part of transitioning to the new encryption scheme for the plurality of data blocks to determine that the one or more data blocks have not been encrypted according to the new encryption scheme, wherein the individual encryption transition state indicates respective ones of a plurality of different schemes currently applied to the plurality of data blocks, and wherein the individual encryption state indicates at least one of the plurality of data blocks currently applies the new encryption scheme when the query is received;
perform in-place encryption for the one or more data blocks so that no database data of the one or more data blocks is moved, wherein to perform the in-place encryption, the database system is configured to:
read the one or more data blocks in the storage system to perform the query;
return a response to the query based on the one or more data blocks
use data read from the one or more data blocks to perform the query to encrypt the data read from the one or more data blocks with the new encryption scheme;
store, in the one or more data blocks, the data encrypted with the new encryption scheme; and
update the individual encryption transition state of the one or more data blocks to indicate that different one of the plurality of schemes, is currently applied.
2 . The system of claim 1 , wherein the database system is further configured to receive a request that specifies the new encryption scheme to be applied.
3 . The system of claim 1 , wherein the database system is further configured to:
detect an encryption window;
identify one or more additional data blocks of the plurality of data blocks to encrypt according to the new encryption scheme based on an evaluation of the metadata;
read the one or more additional data blocks of the from the data storage system;
encrypt the one or more additional data blocks according to the new encryption scheme and the encryption window; and
store the encrypted one or more additional data blocks in the storage system.
4 . The system of claim 1 , wherein the database system is a data warehouse service offered by a provider network.
5 . A method, comprising:
receiving, at a database system, a request that causes one or more data blocks of a plurality of data blocks in a storage system storing a database to be read, wherein the database is in a transition state to a new encryption scheme;
determining, by the database system, that the one or more data blocks have not been encrypted according to the new encryption scheme based, at least in part, on metadata comprising individual encryption transition state maintained as part of transitioning to the new encryption scheme, wherein the individual encryption transition state indicates respective ones of a plurality of different schemes currently applied to the plurality of data blocks, and wherein the individual encryption state indicates at least one of the plurality of data blocks currently applies the new encryption scheme when the request is received;
performing, by the database system, in-place encryption for the one or more data blocks so that no database data of the one or more data blocks is moved, comprising:
reading, by the database system, the one or more data blocks in the storage system to perform the request;
using, by the database system, data read from the one or more data blocks to encrypt the data read from the one or more data blocks with the new encryption scheme;
storing, by the database system in the one or more data blocks, the data encrypted with the new encryption scheme; and
updating the metadata comprising the individual encryption transition state for the one or more data blocks to indicate that the currently applied scheme is the new encryption scheme.
6 . The method of claim 5 , further comprising receiving a request that specifies the new encryption scheme to be applied.
7 . The method of claim 6 , wherein the request that specifies the new encryption scheme is a request to resize a processing cluster implementing the database system.
8 . The method of claim 6 , wherein the request that specifies the new encryption scheme is a request to perform in-place encryption on the database.
9 . The method of claim 5 , wherein the new encryption scheme is applied to a specified table of one or more tables of the database.
10 . The method of claim 5 , further comprising:
detecting, by the database system, an encryption window;
identifying, by the database system, one or more additional data blocks of the plurality of data blocks to encrypt according to the new encryption scheme;
reading, by the database system, the one or more additional data blocks of the from the data storage system;
encrypting, by the database system, the one or more additional data blocks according to the new encryption scheme and the encryption window; and
storing, by the database system, the encrypted one or more additional data blocks in the storage system.
11 . The method of claim 5 , further comprising:
receiving, at the database system, a request to add data to the database;
encrypting, by the database system, the data to add to the database in one or more additional data blocks according to the new encryption scheme; and
storing, by the database system, the one or more additional data blocks in the storage system.
12 . The method of claim 5 , further comprising:
receiving, at the database system, a second request that causes one or more further data blocks of the plurality of data blocks to be read;
determining, by the database system, that the one or more further data blocks have been encrypted according to the new encryption scheme;
reading, by the database system, the one or more further data blocks in the storage system;
decrypting, by the database system, the one or more data blocks with the new encryption scheme; and
performing, by the database system, the second request using the decrypted one or more further data blocks.
13 . One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement:
receiving a query that causes one or more data blocks of a plurality of data blocks in a storage system storing a database to be read, wherein the database is in a transition state to a new encryption scheme;
determining that the one or more data blocks have not been encrypted according to the new encryption scheme based, at least in part, on metadata comprising individual encryption transition state maintained as part of transitioning to the new encryption scheme, wherein the individual encryption transition state indicates respective ones of a plurality of different schemes currently applied to the plurality of data blocks, and wherein the individual encryption state indicates at least one of the plurality of data blocks currently applies the new encryption scheme when the query is received;
performing in-place encryption for the one or more data blocks so that no database data of the one or more data blocks is moved, wherein, in performing the in-place encryption, the program instructions cause the one or more computing devices to implement:
reading the one or more data blocks in the storage system to perform the query;
using data read from the one or more data blocks to encrypt the data read from the one or more data blocks with the new encryption scheme;
storing, in the one or more data blocks, the data encrypted with the new encryption scheme; and
updating the metadata comprising the individual encryption transition state for the one or more data blocks to indicate that the currently applied scheme is the new encryption scheme.
14 . The one or more non-transitory, computer-readable storage media of claim 13 , storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement receiving a request that specifies the new encryption scheme to be applied as part of a restore operation for the database, wherein the plurality of data blocks are stored as part of a backup copy of the database.
15 . The one or more non-transitory, computer-readable storage media of claim 13 , storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement receiving a request that specifies the new encryption scheme to be applied as part of in-place encryption on the database.
16 . The one or more non-transitory, computer-readable storage media of claim 13 , wherein the new encryption scheme is applied to a specified column of one or more columns of a table of the database.
17 . The one or more non-transitory, computer-readable storage media of claim 13 , storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement:
detecting an encryption window;
identifying one or more additional data blocks of the plurality of data blocks to encrypt according to the new encryption scheme;
reading the one or more additional data blocks of the from the data storage system;
encrypting the one or more additional data blocks according to the new encryption scheme and the encryption window; and
storing the encrypted one or more additional data blocks in the storage system.
18 . The one or more non-transitory, computer-readable storage media of claim 13 , storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement:
receiving, at the database system, a request to add data to the database;
encrypting, by the database system, the data to add to the database in one or more additional data blocks according to the new encryption scheme; and
storing, by the database system, the one or more additional data blocks in the storage system.
19 . The one or more non-transitory, computer-readable storage media of claim 13 , wherein the one or more computing devices are implemented as part of a database service offered by a provider network.