Log-structured distributed storage using a single log sequence number space
A distributed database system may implement log-structured distributed storage using a single log sequence number space. A log for a data volume may be maintained in a log-structured distributed storage system. The log may be segmented across multiple protection groups according to a partitioning of user data for the data volume. Updates to the log may be assigned a log sequence number from a log sequence number space for the data volume. A protection group may be determined for an update according to which partition of user data space the update pertains. Metadata to be included with the log record may indicate a previous log sequence number of a log record maintained at the protection group. The log record may be sent to the protection group and identified as committed based on acknowledgments received from storage nodes implementing the protection group.
1. A system, comprising:
a memory to store program instructions which, if performed by at least one processor, cause the at least one processor to perform a method to at least:
detect a failure of a storage client of a distributed data store that stores a log for a data volume, wherein the log is segmented across a plurality of different groups of storage nodes that store different partitions of the data volume, wherein at least one group of storage nodes provides redundant storage for a given segment of the log, and wherein the log comprises a plurality of log records assigned respective log sequence numbers according to a log sequence number space for the data volume; and
obtain from each of the plurality of groups one or more log records that indicate possible recovery points for the log;
evaluate the assigned log sequence numbers for the obtained log records to identify a recovery point in the log for the data volume according to the log sequence number space; and
make a view of the data volume available for access requests consistent with the recovery point in the log for the data volume.
2. The system of claim 1 ,
wherein to obtain from each of the plurality of groups one or more log records that indicate possible recovery points for the log, comprises send a request for the one or more log records that indicate the possible recovery points for the logs to at least one storage node in each of the groups of storage nodes;
wherein the method further comprises:
for each of the groups of storage nodes:
receive the request at the at least one storage node;
based, at least in part, on metadata indicating a previous log sequence number for log records maintained at the group of storage nodes, select, by the at least one storage node, one or more candidate log records of the log records maintained at the group of storage nodes according to a recovery sequence for the log records maintained at the group of storage nodes; and
send, from the at least one storage node, respective metadata for the one or more candidate log records in response to the request for the one or more log records.
3. The system of claim 2 , wherein the method further comprises:
for each of the groups of storage nodes, remove, by the at least one storage node, at least one of the candidate log records that do not indicate completion of a system transaction.
4. The system of claim 2 , wherein to select the one or more candidate log records of the log records comprises traverse one or more links between the log records in the metadata to locate a first break in the recovery sequence.
5. The system of claim 2 , wherein the selection of the one or more candidate log records is based, at least in part, on a volume-wide recovery point in the metadata.
6. The system of claim 1 , wherein to evaluate the assigned log sequence numbers for the obtained log records to identify the recovery point comprises determine a break in the log sequence number space for the obtained log records.
7. The system of claim 1 , wherein the memory and the at least one processor are implemented as part of a recovery service for a network-based database service.
8. A method, comprising:
detecting a failure of a storage client of a distributed data store that stores a log for a data volume, wherein the log is segmented across a plurality of different groups of storage nodes that store different partitions of the data volume, wherein at least one group of storage nodes provides redundant storage for a given segment of the log, and wherein the log comprises a plurality of log records assigned respective log sequence numbers according to a log sequence number space for the data volume; and
obtaining from each of the plurality of groups one or more log records that indicate possible recovery points for the log;
evaluating the assigned log sequence numbers for the obtained log records to identify a recovery point in the log for the data volume according to the log sequence number space; and
making a view of the data volume available for access requests consistent with the recovery point in the log for the data volume.
9. The method of claim 8 ,
wherein the obtaining from each of the plurality of groups one or more log records that indicate possible recovery points for the log, comprises sending a request for the one or more log records that indicate the possible recovery points for the logs to at least one storage node in each of the groups of storage nodes;
wherein the method further comprises:
for each of the groups of storage nodes:
receiving the request at the at least one storage node;
based, at least in part, on metadata indicating a previous log sequence number for log records maintained at the group of storage nodes, selecting, by the at least one storage node, one or more candidate log records of the log records maintained at the group of storage nodes according to a recovery sequence for the log records maintained at the group of storage nodes; and
sending, from the at least one storage node, respective metadata for the one or more candidate log records in response to the request for the one or more log records.
10. The method of claim 9 , further comprising:
for each of the groups of storage nodes, removing, by the at least one storage node, at least one of the candidate log records that do not indicate completion of a system transaction.
11. The method of claim 9 , wherein the selecting the one or more candidate log records of the log records comprises traversing one or more links between the log records in the metadata to locate a first break in the recovery sequence.
12. The method of claim 9 , wherein the selecting of the one or more candidate log records is based, at least in part, on a volume-wide recovery point in the metadata.
13. The method of claim 8 , wherein the evaluating the assigned log sequence numbers for the obtained log records to identify the recovery point comprises determining a break in the log sequence number space for the obtained log records.
14. The method of claim 8 , wherein the detecting, the obtaining, the evaluating and the making are performed by a recovery service for a network-based database service.
15. A non-transitory, computer-readable storage medium, comprising program instructions that when executed by at least one computing device cause the at least one computing device to implement:
detecting a failure of a storage client of a distributed data store that stores a log for a data volume, wherein the log is segmented across a plurality of different groups of storage nodes that store different partitions of the data volume, wherein at least one group of storage nodes provides redundant storage for a given segment of the log, and wherein the log comprises a plurality of log records assigned respective log sequence numbers according to a log sequence number space for the data volume; and
obtaining from each of the plurality of groups one or more log records that indicate possible recovery points for the log;
evaluating the assigned log sequence numbers for the obtained log records to identify a recovery point in the log for the data volume according to the log sequence number space; and
making a view of the data volume available for access requests consistent with the recovery point in the log for the data volume.
16. The non-transitory, computer-readable storage medium of claim 15 ,
wherein, in obtaining from each of the plurality of groups one or more log records that indicate possible recovery points for the log, the program instructions cause the at least one computing device to implement sending a request for the one or more log records that indicate the possible recovery points for the logs to at least one storage node in each of the groups of storage nodes;
wherein the program instructions cause the at least one computing device to further implement:
for each of the groups of storage nodes:
receiving the request at the at least one storage node;
based, at least in part, on metadata indicating a previous log sequence number for log records maintained at the group of storage nodes, selecting, by the at least one storage node, one or more candidate log records of the log records maintained at the group of storage nodes according to a recovery sequence for the log records maintained at the group of storage nodes; and
sending, from the at least one storage node, respective metadata for the one or more candidate log records in response to the request for the one or more log records.
17. The non-transitory, computer-readable storage medium of claim 16 , wherein the program instructions further cause the at least one computing device to implement:
for each of the groups of storage nodes, removing, by the at least one storage node, at least one of the candidate log records that do not indicate completion of a system transaction.
18. The non-transitory, computer-readable storage medium of claim 16 , wherein, in selecting the one or more candidate log records of the log records, the program instructions cause the at least one computing device to implement traversing one or more links between the log records in the metadata to locate a first break in the recovery sequence.
19. The non-transitory, computer-readable storage medium of claim 15 , wherein, in evaluating the assigned log sequence numbers for the obtained log records to identify the recovery point, the program instructions cause the at least one computing device to implement determining a break in the log sequence number space for the obtained log records.
20. The non-transitory, computer-readable storage medium of claim 15 , wherein the at least one computing device is implemented as part of a recovery service for a network-based database service.