Database system with database engine and separate distributed storage service
A database system may include a database service and a separate distributed storage service. The database service (or a database engine head node thereof) may be responsible for query parsing, optimization, and execution, transactionality, and consistency, while the storage service may be responsible for generating data pages from redo log records and for durability of those data pages. For example, in response to a write request directed to a particular data page, the database engine head node may generate a redo log record and send it, but not the data page, to a storage service node. The storage service node may store the redo log record and return a write acknowledgement to the database service prior to applying the redo log record. The server node may apply the redo log record and other redo log records to a previously stored version of the data page to create a current version.
1 . A system, comprising:
at least one processor; and
a memory, storing program instructions that when executed by the at least one processor cause the at least one processor to:
receive, from a client, a write request to add data to a database table;
in response to the reception of the write request, send an indication of the write request to a plurality of storage nodes of a distributed storage system that store at least a portion of the database table;
receive, from a number of the plurality of storage nodes of the distributed storage system that satisfies a write quorum, an indication for performance of the write request; and
responsive to reception of the indication for performance of the write request from the number of the plurality of storage nodes that satisfies the write quorum, return to the client an acknowledgement of the write request as complete.
2 . The system of claim 1 , further comprising program instructions that when executed by the at least one processor cause the at least one processor to:
receive, from the client, a different write request to add other data to the database table;
in response to the reception of the different write request, send an indication of the different write request to the plurality of storage nodes of the distributed storage system; and
after a period of time without receiving enough write acknowledgements from the number of the plurality of storage nodes that satisfies the write quorum, determine that an error condition exists in the distributed storage system.
3 . The system of claim 1 , wherein a given storage node comprises a virtual machine on which storage node server code is deployed.
4 . The system of claim 1 , further comprising program instructions that when executed by the at least one processor cause the at least one processor to:
receive, from the client, a read request directed to a data page; and
responsive to a determination that a cache comprises the data page, return, to the client, data corresponding to the read request.
5 . The system of claim 1 , wherein the plurality of storage nodes of the distributed storage system store data for a network-based database service.
6 . The system of claim 1 , wherein the plurality of storage nodes comprises a protection group for a portion of a database volume.
7 . The system of claim 6 , wherein the database volume comprises a contiguous sequence of volume extents.
8 . A method, comprising:
receiving, from a client, a write request to add data to a database table;
in response to receiving the write request, sending an indication of the write request to a plurality of storage nodes of a distributed storage system that store at least a portion of the database table;
receiving, from a number of the plurality of storage nodes of the distributed storage system that satisfies a write quorum, an indication for performance of the write request; and
responsive to receiving the indication for performance of the write request from the number of the plurality of storage nodes that satisfies the write quorum, returning to the client an acknowledgement of the write request as complete.
9 . The method of claim 8 , further comprising:
receiving, from the client, a different write request to add other data to the database table;
in response to receiving the different write request, sending an indication of the different write request to the plurality of storage nodes of the distributed storage system; and
after a period of time without receiving enough write acknowledgements from the number of the plurality of storage nodes that satisfies the write quorum, determining that an error condition exists in the distributed storage system.
10 . The method of claim 8 , wherein a given storage node comprises a virtual machine on which storage node server code is deployed.
11 . The method of claim 8 , further comprising:
receiving, from the client, a read request directed to a data page; and
responsive to determining that a cache comprises the data page, returning, to the client, data corresponding to the read request.
12 . The method of claim 8 , wherein the plurality of storage nodes of the distributed storage system store data for a network-based database service.
13 . The method of claim 8 , wherein the plurality of storage nodes comprises a protection group for a portion of a database volume.
14 . The method of claim 13 , wherein the database volume comprises a contiguous sequence of volume extents.
15 . The system of claim 1 , further comprising program instructions that when executed) One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices, cause the one or more computing devices to implement:
receiving, from a client, a write request to add data to a database table;
in response to receiving the write request, sending an indication of the write request to a plurality of storage nodes of a distributed storage system that store at least a portion of the database table;
receiving, from a number of the plurality of storage nodes of the distributed storage system that satisfies a write quorum, an indication for performance of the write request; and
responsive to receiving the indication for performance of the write request from the number of the plurality of storage nodes that satisfies the write quorum, returning to the client an acknowledgement of the write request as complete.
16 . The one or more non-transitory, computer-readable storage media of claim 15 , storing further instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement:
receiving, from the client, a different write request to add other data to the database table;
in response to receiving the different write request, sending an indication of the different write request to the plurality of storage nodes of the distributed storage system; and
after a period of time without receiving enough write acknowledgements from the number of the plurality of storage nodes that satisfies the write quorum, determining that an error condition exists in the distributed storage system.
17 . The one or more non-transitory, computer-readable storage media of claim 15 , wherein a given storage node comprises a virtual machine on which storage node server code is deployed.
18 . The one or more non-transitory, computer-readable storage media of claim 15 , storing further instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement:
receiving, from the client, a read request directed to a data page; and
responsive to determining that a cache comprises the data page, returning, to the client, data corresponding to the read request.
19 . The one or more non-transitory, computer-readable storage media of claim 15 , wherein the plurality of storage nodes of the distributed storage system store data for a network-based database service.
20 . The one or more non-transitory, computer-readable storage media of claim 15 , wherein the plurality of storage nodes comprises a protection group for a portion of a database volume.