Recovery from a system fault in a cloud-based storage system
Recovery in a cloud-based storage system, including: receiving, by the cloud-based storage system among a plurality of storage systems synchronously replicating a dataset, a request to modify the dataset; generating recovery information indicating whether the request to modify the dataset has been applied on all storage systems in the plurality of storage systems synchronously replicating the dataset; and responsive to a system fault, applying a recovery action in dependence upon the recovery information indicating whether the request to modify the dataset has been applied on all storage systems in the plurality of storage systems synchronously replicating the dataset.
1. A method comprising:
receiving, by a cloud computing environment implementing a cloud-based storage system, a client request to modify a dataset being replicated among a plurality of storage systems including the cloud-based storage system, wherein each storage system is configured to individually receive and process the client request to modify the dataset, and wherein each storage system comprises a plurality of storage controllers with each storage controller interconnected to a plurality of storage devices;
generating recovery information indicating whether the client request to modify the dataset has been applied on all the storage systems in the plurality of storage systems; and
after a system fault at one of the plurality of storage systems, applying a recovery action at least one storage system of the plurality of storage systems based on the recovery information indicating whether the client request to modify the dataset has been applied on all storage systems in the plurality of storage systems.
2. The method of claim 1 , wherein the cloud-based storage system includes a plurality of cloud computing instances that respectively store a portion of the dataset.
3. The method of claim 2 , wherein the cloud computing instances provide block-level data storage.
4. The method of claim 1 , wherein the cloud-based storage system includes a cloud-based object storage that stores all portions of the dataset.
5. The method of claim 1 , wherein each storage controller includes computer program instructions executing on a cloud computing instance.
6. The method of claim 1 , wherein the recovery information includes tracking information from one or more storage systems of the plurality of storage systems, and wherein the tracking information indicates progress toward applying the client request to modify the dataset on the plurality of storage systems.
7. The method of claim 6 , wherein the tracking information includes logging information on which storage systems of the plurality of storage systems have been sent the client request to modify the dataset.
8. The method of claim 6 , wherein the tracking information includes logging information on which storage systems of the plurality of storage systems have acknowledged completing the client request to modify the dataset.
9. The method of claim 1 , wherein the recovery information includes confirmation information from one or more other of the plurality of storage systems on whether the client request is confirmed to have been processed, and wherein the confirmation information is based on the cloud-based storage system querying the one or more of the other of the plurality of storage systems.
10. The method of claim 1 , wherein the client request to modify the dataset is one of a plurality of client requests to modify the dataset, and wherein the recovery information comprises an order in which the plurality of client requests to modify the dataset are to be applied.
11. A cloud-based storage system including:
one or more cloud computing instances providing a first tier of cloud storage;
a second tier of cloud storage; and
one or more storage controller applications, each storage controller application executing in a cloud computing instance, wherein the one or more storage controllers are configured to:
receive, by at least one of the one or more storage controller applications, a client request to modify a dataset being replicated among a plurality of storage systems including the cloud-based storage system, wherein each storage system is configured to individually receive and process the client request to modify the dataset, and wherein each storage system comprises a plurality of storage controllers with each storage controller interconnected to a plurality of storage devices;
generate recovery information indicating whether the client request to modify the dataset has been applied on all the storage systems in the plurality of storage systems; and
responsive to a system fault at one of the plurality of the storage systems, apply a recovery action at at least one storage system of the plurality of storage systems in dependence upon the recovery information indicating whether the client request to modify the dataset has been applied on all storage systems in the plurality of storage systems.
12. The cloud-based storage system of claim 11 , wherein the first tier of cloud storage of the cloud-based storage system stores a portion of the dataset.
13. The cloud-based storage system of claim 11 , wherein the second tier of cloud storage of the cloud-based storage system stores all portions of the dataset.
14. The cloud-based storage system of claim 11 , wherein the second tier of cloud storage of the cloud-based storage system includes a cloud object store.
15. The cloud-based storage system of claim 11 , wherein the plurality of storage systems includes at least one hardware-based storage system.
16. The cloud-based storage system of claim 11 , wherein the recovery information includes tracking information from one or more storage systems of the plurality of storage systems, and wherein the tracking information indicates progress toward applying the client request to modify the dataset on the plurality of storage systems.
17. The cloud-based storage system of claim 16 , wherein the tracking information includes logging information on which storage systems of the plurality of storage systems have been sent the client request to modify the dataset.
18. The cloud-based storage system of claim 16 , wherein the tracking information includes logging information on which storage systems of the plurality of storage systems have acknowledged completing the client request to modify the dataset.
19. The cloud-based storage system of claim 11 , wherein the recovery information includes confirmation information from one or more other of the plurality of storage systems on whether the client request is confirmed to have been processed, and wherein the confirmation information is based on the cloud-based storage system querying the one or more of the other of the plurality of storage systems.
20. The cloud-based storage system of claim 11 , wherein the client request to modify the dataset is one of a plurality of client requests to modify the dataset, and wherein the recovery information comprises an order in which the plurality of client requests to modify the dataset are to be applied.