Recovering from system faults for replicated datasets
Recovering from system faults for replicated datasets, including: receiving, by the cloud-based storage system, a request to modify a dataset that is stored by the cloud-based storage system, wherein the dataset is synchronously replicated among a plurality of storage systems that includes the cloud-based storage system, wherein a request to modify the dataset is acknowledged as being complete when each of the plurality of storage systems has modified its copy of the dataset; generating recovery information indicating whether the request to modify the dataset has been applied on all storage systems in the plurality of storage systems synchronously replicating the dataset; and after a system fault, applying a recovery action in dependence upon the recovery information indicating whether the request to modify the dataset has been applied on all storage systems in the plurality of storage systems synchronously replicating the dataset.
1. A method comprising:
receiving, by a cloud-based storage system, a request to modify a dataset that is stored by the cloud-based storage system, wherein the dataset is synchronously replicated among a plurality of storage systems that includes the cloud-based storage system;
generating recovery information indicating whether the request to modify the dataset has been applied on all storage systems in the plurality of storage systems, wherein the plurality of storage systems is synchronously replicating the dataset by acknowledging the request as being complete when each storage system of the plurality of storage systems has modified its copy of the dataset; and
after a system fault, applying a recovery action based on the recovery information indicating whether the request to modify the dataset has been applied on all storage systems in the plurality of storage systems synchronously replicating the dataset.
2. The method of claim 1 , wherein the cloud-based storage system includes a plurality of cloud computing instances that respectively store a portion of the dataset.
3. The method of claim 2 , wherein the cloud computing instances provide block-level data storage.
4. The method of claim 1 , wherein the cloud-based storage system includes a cloud-based object storage that stores all portions of the dataset.
5. The method of claim 1 , wherein the plurality of storage systems includes at least one hardware-based storage system.
6. The method of claim 1 , wherein the recovery information includes tracking information from one or more of the plurality of storage systems, and wherein the tracking information indicates progress toward applying the request to modify the dataset on the plurality of storage systems.
7. The method of claim 6 , wherein the tracking information includes logging information on which storage systems of the plurality of storage systems have been sent the request to modify the dataset.
8. The method of claim 6 , wherein the tracking information includes logging information on which storage systems of the plurality of storage systems have acknowledged completing the request to modify the dataset.
9. The method of claim 1 , wherein the recovery information includes confirmation information from one or more other of the plurality of storage systems on whether the request is confirmed to have been processed, and wherein the confirmation information is based on the cloud-based storage system querying the one or more of the other of the plurality of storage systems.
10. The method of claim 1 , wherein the request to modify the dataset is one of a plurality of requests to modify the dataset, and wherein the recovery information comprises an order in which the plurality of requests to modify the dataset are to be applied.
11. A cloud-based storage system including:
one or more storage controller applications, each storage controller application executing in a cloud computing instance, wherein the one or more storage controller applications are configured to:
receive a request to modify a dataset that is stored by the cloud-based storage system, wherein the dataset is synchronously replicated among a plurality of storage systems that includes the cloud-based storage system,
generate recovery information indicating whether the request to modify the dataset has been applied on all storage systems in the plurality of storage systems, wherein the plurality of storage systems is synchronously replicating the dataset by acknowledging the request as being complete when each storage system of the plurality of storage systems has modified its copy of the dataset; and
after a system fault, apply a recovery action based on the recovery information indicating whether the request to modify the dataset has been applied on all storage systems in the plurality of storage systems synchronously replicating the dataset.
12. The cloud-based storage system of claim 11 , wherein a first tier of cloud storage of the cloud-based storage system stores a portion of the dataset, wherein the first tier includes block storage.
13. The cloud-based storage system of claim 12 , wherein a second tier of cloud storage of the cloud-based storage system stores all portions of the dataset, wherein the second tier includes object storage.
14. The cloud-based storage system of claim 11 , wherein the first tier of cloud storage includes a plurality of cloud computing instances with local storage, and each cloud computing instance that executes a storage controller application is communicatively coupled to each of the cloud computing instances with local storage.
15. The cloud-based storage system of claim 11 , wherein the plurality of storage systems includes at least one hardware-based storage system.
16. The cloud-based storage system of claim 11 , wherein the recovery information includes tracking information from one or more of the plurality of storage systems, and wherein the tracking information indicates progress toward applying the request to modify the dataset on the plurality of storage systems.
17. The cloud-based storage system of claim 16 , wherein the tracking information includes logging information on which storage systems of the plurality of storage systems have been sent the request to modify the dataset.
18. The cloud-based storage system of claim 16 , wherein the tracking information includes logging information on which storage systems of the plurality of storage systems have acknowledged completing the request to modify the dataset.
19. The cloud-based storage system of claim 11 , wherein the recovery information includes confirmation information from one or more other of the plurality of storage systems on whether the request is confirmed to have been processed, and wherein the confirmation information is based on the cloud-based storage system querying the one or more of the other of the plurality of storage systems.
20. The cloud-based storage system of claim 11 , wherein the request to modify the dataset is one of a plurality of requests to modify the dataset, and wherein the recovery information comprises an order in which the plurality of requests to modify the dataset are to be applied.