Asynchronous data set updates for accessing database partitions
An update to a data set is received at a distributed database system. The data set is stored with or cached for accessing different partitions of a database. A new version identifier of the data set is written to indicate the update to the data set using a corresponding log record written to respective update logs of the different partitions of the database. The new version identifier is then used as part of subsequent validation operations for access requests to the database.
1 . A system, comprising:
a plurality of computing devices, respectively comprising at least one processor and a memory, that implement a distributed database system, wherein the distributed database system is configured to:
receive a request to modify a schema of a database table, wherein the database table is stored in different partitions across the plurality of computing devices;
update respective copies of metadata of a catalog, wherein the catalog stores the schema of the database table, wherein the respective copies of the metadata of the catalog are co-located with individual ones of the different partitions, wherein to update the respective copies of the metadata of the catalog, the distributed database system writes a new version identifier of the metadata of the catalog with a corresponding log record that includes the modification to the catalog at respective update logs of the different partitions of the database table; and
use the new version identifier as input to a validation operation for different operations that use the metadata of the catalog as part of accessing individual ones of different partitions of the database table, wherein access requests to the database table are rejected that are determined to be invalid by the validation operation, wherein one of the access requests is a request to write data in at least one of the partitions, wherein the one access request succeeds according to a determination that an assumed version of the data set is valid with respect to the new version of the data set as part of the validation operation.
2 . The system of claim 1 , wherein the distributed database system uses a commitment component, separate from a storage system that stores the respective copies of the metadata of the catalog and the different partitions of the database table, to commit the update to the schema using the respective copies of the update log.
3 . The system of claim 1 , wherein the distributed database system comprises a plurality of query processors that cache the metadata of the catalog to perform different ones of the different access requests.
4 . The system of claim 1 , wherein the distributed database system is a database service implemented across a plurality of regions of a provider network.
5 . A method, comprising:
receiving, at a distributed database system, an update to a data set stored in respective copies along with different partitions of a database;
performing, by the distributed database system, the update to the data set, comprising writing a new version identifier of the data set with a corresponding log record that includes the update to the data set at respective update logs of the different partitions of the database; and
using, by the distributed database system, the new version identifier as input to a validation operation for different operations that access the data set as part of accessing individual ones of different partitions of the database, wherein access requests to the database are rejected that are determined to be invalid by the validation operation, wherein one of the access requests is a request to write data in at least one of the partitions, wherein the one access request succeeds according to a determination that an assumed version of the data set is valid with respect to the new version of the data set as part of the validation operation.
6 . The method of claim 5 , wherein the distributed database system uses a commitment component, separate from a storage system that stores the respective copies of the data set and the different partitions of the database, to commit the update to the data set using the respective copies of the update log.
7 . The method of claim 5 , wherein a different one of the access requests is a request to read data in at least one of the partitions, wherein the one access request succeeds according to a determination that an assumed version of the data set is valid with respect to the new version of the data set as part of the validation operation.
8 . The method of claim 5 , wherein a different one of the access requests is a request to write data in at least one of the partitions, wherein the one access request is rejected according to a determination that an assumed version of the data set is not valid with respect to the new version of the data set as part of the validation operation.
9 . The method of claim 5 , wherein access requests to the database that are rejected include an indication of the new version of the data set as part of an error response.
10 . The method of claim 5 , further comprising:
receiving a further update to the data set; and
rejecting the further update to the data set responsive to a determination that the update is to the data set is not complete.
11 . The method of claim 5 , wherein the update to the data set is an update to a schema, wherein the data set is metadata of a catalog cached at a plurality of different query processors that independently perform different ones of the access requests to the database.
12 . The method of claim 5 , wherein the partitions of the database are partitions of one or more relational database tables.
13 . One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement:
receiving, at a distributed database system, an update to a data set stored in respective copies along with different partitions of a database;
performing, by the distributed database system, the update to the data set, comprising writing a new version identifier of the data set with a corresponding log record that includes the update to the data set at respective update logs of the different partitions of the database; and
using, by the distributed database system, the new version identifier as input to a validation operation for different operations that access the data set as part of accessing individual ones of different partitions of the database, wherein access requests to the database are rejected that are determined to be invalid by the validation operation, wherein one of the access requests is a request to write data in at least one of the partitions, wherein the one access request succeeds according to a determination that an assumed version of the data set is valid with respect to the new version of the data set as part of the validation operation.
14 . The one or more non-transitory, computer-readable storage media of claim 13 , wherein the distributed database system uses a commitment component, separate from a storage system that stores the respective copies of the data set and the different partitions of the database, to commit the update to the data set using the respective copies of the update log.
15 . The one or more non-transitory, computer-readable storage media of claim 13 , wherein a different one of the access requests is a request to read data in at least one of the partitions, wherein the different one of the access requests succeeds according to a determination that an assumed version of the data set is valid with respect to the new version of the data set as part of the validation operation.
16 . The one or more non-transitory, computer-readable storage media of claim 13 , storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement:
receiving a further update to the data set; and
rejecting the further update to the data set responsive to a determination that the update is to the data set is not complete.
17 . The one or more non-transitory, computer-readable storage media of claim 13 , wherein the update to the data set is an update to a schema, wherein the data set is catalog metadata cached at a plurality of different query processors that independently perform different ones of the access requests to the database.
18 . The one or more non-transitory, computer-readable storage media of claim 13 , wherein the distributed database system is a database service implemented across a plurality of regions of a provider network.