Hierarchical authority store for cross-zone replicated block storage devices
The present disclosure generally relates to creating virtualized block storage devices whose data is replicated across isolated computing systems to lower risk of data loss even in wide-scale events, such as natural disasters. The virtualized device can include at least two volumes, each of which is implemented in a distinct computing system. Each volume can be implemented by at least two computing devices, a first of which is configured as a primary device to which reads from and writes to the volume are directed. To ensure consistency in the distributed device, a multi-tier authority service is implemented, in which a cross-computing system authority service designates a volume as having authority to accept writes to the virtualized device, and in which a second tier authority service designates a computing device as having authority to accept writes to the volume.
1. A system for providing a data store that is replicated across at least first and second zones of a cloud provider network, wherein the first and second zones are in communication via a communication network but are electrically and physically isolated from one another at least partly by being geographically distanced from one another and having independent electrical power, wherein the data store comprises a first replicated data volume in the first zone and a second replicated data volume in the second zone, the system comprising:
a first computing system associated with the first zone, the first computing system including at least a first computing device and a second computing device that collectively implement the first replicated data volume to provide a first copy of the data of the data store by providing redundant endpoints for reading from and writing to the first replicated data volume, wherein the first computing system further includes an authority service device configured to store a designation of the first computing device as having authority to receive writes to the first replicated data volume; and
a second computing system associated with the second zone, the second computing system including at least two computing devices that collectively implement the second replicated data volume to provide a second copy of the data of the data store by providing additional redundant endpoints for reading from and writing to the second replicated data volume, wherein the second replicated data volume is configured to replicate the writes from the first replicated data volume; and
one or more computing devices implementing a multi-zone authority service configured to store a designation of the first computing system as having authority to receive writes to the data store;
wherein the first computing device is configured, prior to accepting the writes to the data store from a client device and storing the writes on the first replicated data volume, to verify that i) the multi-zone authority service stores the designation of the first zone as having authority to receive writes to the data store and ii) the authority service device of the first computing system stores the designation of the first computing device as having authority to receive writes to the first replicated data volume.
2. The system of claim 1 , wherein the client device is a virtual machine instance, and wherein first and second replicated data volumes represent a virtualized block storage device of the virtual machine instance.
3. The system of claim 1 , wherein first computing device is further configured to obtain reads of the first replicated data volume from the client device and to respond to the reads by providing data stored at the first computing device after verifying that i) the multi-zone authority service stores the designation of the first computing system as having authority to receive writes to the data store and ii) the authority service device of the first computing system stores the designation of the first computing device as having authority to receive writes to the first replicated data volume.
4. The system of claim 1 , wherein the write is stored at the first computing device within a write journal, and wherein the first computing device is further configured to persist the write to a physical storage device.
5. The system of claim 1 , wherein first computing device persists the write to a physical storage device using erasure coding.
6. A computer-implemented method to provide redundancy in a virtualized storage device replicated across at least first and second zones, wherein the first and second zones are in communication via a communication network but are isolated from one another, wherein the virtualized storage device comprises a first volume in the first zone and a second volume in the second zone, each of the first and second volumes being implemented by at least two computing devices in the respective first and second zones that collectively operate to provide redundant endpoints for reading from and writing to the volumes in the respective first and second zones, the computer-implemented method comprising, at a first computing device of the at least two computing devices implementing the first volume:
obtaining a request to write data to the virtualized storage device;
prior to accepting the request:
verifying that an authority service within the first zone stores a designation of the first computing device as having authority to receive writes to the first volume;
verifying that an authority service distributed among the first and second zone stores information usable to identify the first volume as having authority to receive writes to the virtualized storage device; and
in response to verifying that i) the authority service within the first zone stores the designation of the first computing device as having authority to receive writes to the first volume, and ii) the authority service distributed among the first and second zone stores information usable to identify the first volume as having authority to receive writes to the virtualized storage device, accepting the request at least partly by:
storing the write at the first computing device;
replicating the write to both i) a second computing device of the at least two computing devices implementing the first volume and ii) the second volume; and
acknowledging the request.
7. The computer-implemented method of claim 6 , wherein verifying that the authority service within the first zone stores the designation of the first computing device as having authority to receive writes to the first volume comprises verifying that the first computing device holds a valid lease from the authority service within the first zone designating the first computing device as having authority to receive writes to the first volume.
8. The computer-implemented method of claim 7 further comprising periodically renewing the lease with the authority service within the first zone.
9. The computer-implemented method of claim 6 , wherein the authority service is further distributed among a third zone isolated from the first and second zones, and wherein the method further comprises, at the authority service:
obtaining from the first computing device a request for a lease designating the first computing device as having authority to receive writes to the first volume;
determining that a majority of devices implementing the authority service among the first, second, and third zones designate the first computing device as having authority to receive writes to the first volume; and
returning to the first computing device the lease.
10. The computer-implemented method of claim 9 , wherein determining that the majority of devices implementing the authority service among the first, second, and third zones designate the first computing system as having authority to receive writes to the first volume comprises implementing a consensus protocol among the devices implementing the authority service.
11. The computer-implemented method of claim 6 , wherein acknowledging the request occurs only after replicating the write to both i) the second computing device and ii) the second volume.
12. The computer-implemented method of claim 6 , wherein acknowledging the request occurs after obtaining acknowledgement of the write from the second computing device and without awaiting acknowledgement of the write from the second volume.
13. A system to provide redundancy in a virtualized storage device replicated across at least a first and second zone, wherein the virtualized storage device comprises a first volume in the first zone and a second volume in the second zone, each of the first and second volumes being implemented by at least two computing devices in the respective first and second zones that collectively operate to provide redundant endpoints for reading from and writing to the volumes in the respective zones, the system comprising:
a first computing device of the at least two computing devices implementing the first volume, the first computing device configured to:
obtain a request to write data to the virtualized storage device;
verify that an authority service within the first zone stores a designation of the first computing system as having authority to receive writes to the first volume;
verify that an authority service distributed among the first and second zones stores information usable to identify the first volume as having authority to receive writes to the virtualized storage device; and
subsequent to verification that i) the authority service within the first zone stores the designation of the first computing system as having authority to receive writes to the first volume and that ii) the authority service distributed among the first and second zone stores the designation of the first volume as having authority to receive writes to the virtualized storage device, writing the data to the virtualized storage device.
14. The system of claim 13 , wherein to verify that the authority distributed among the first and second zones stores the designation of the first volume as having authority to receive writes to the virtualized storage device, the first computing device is configured to validate a lease from the authority service distributed among the first and second zone designating the first computing system as having authority to receive writes to the first volume.
15. The system of claim 14 , wherein the first computing device is further configured to periodically renew the lease with the authority service distributed among the first and second zones.
16. The system of claim 13 further comprising the authority service distributed among the first and second zones, wherein the authority service distributed among the first and second zones comprises a plurality of computing devices configured to:
obtain a request for designation of the first volume as having authority to receive writes to the virtualized storage device;
implement a consensus protocol to determine that the first volume has authority to receive writes to the virtualized storage device; and
return a lease designating the first volume as having authority to receive writes to the virtualized storage device.
17. The system of claim 16 , wherein the lease is associated with a duration during which the lease is valid.
18. The system of claim 13 , wherein the first computing device is further configured to:
obtain a second request to write data to the virtualized storage device;
determine that the authority service within the first zone does not store the designation of the first computing system as having authority to receive writes to the first volume; and
decline the second request.
19. The system of claim 13 , wherein the information usable to identify the first volume as having authority to receive writes to the virtualized storage device is a designation of the first zone as containing a primary volume of the virtualized storage device.
20. The system of claim 13 wherein the first computing device is further configured to periodically transmit to a control plane device performance metrics of the first computing device.