Synchronously replicating datasets in cloud-based storage systems
A storage system for synchronously replicating a dataset across hardware-based storage systems and cloud-based storage systems, the storage system including specifying a set of managed objects, a set of management operations, and a set of access operations to correspond to the dataset, where management operations can modify or query managed objects equivalently through the hardware-based storage systems and the cloud-based storage systems; access operations to read or modify the dataset operate equivalently through the hardware-based storage systems and the cloud-based storage systems; each hardware-based storage system and each cloud-based storage system stores a separate copy of the dataset as a proper subset of the datasets stored and advertised for use; and operations to modify managed objects or the dataset performed and completed through the hardware-based storage systems and the cloud-based storage systems are reflected in subsequent management objects to query the dataset or subsequent access operations to read the dataset.
1. A storage system including memory storing program instructions that, when executed, cause the storage system to carry out the steps of:
specifying a set of managed objects, a set of management operations, and a set of access operations to correspond to a dataset that is synchronously replicated across hardware-based storage systems and cloud-based storage systems;
detecting a disruption in data communications with one or more of the other hardware-based storage systems or cloud-based storage systems;
determining whether the particular storage system should continue to synchronously replicate the dataset;
responsive to determining that the particular storage system should continue to synchronously replicate the dataset, keeping the dataset on the particular storage system accessible for management and dataset operations; and
responsive to determining that the particular storage system should not continue to synchronously replicate the dataset, making the dataset on the particular storage system inaccessible for management and dataset operations.
2. The storage system of claim 1 , wherein the program instructions, when executed, cause the storage system to carry out the step of:
receiving a request to read a portion of the dataset; and
processing the request to read the portion of the dataset locally.
3. The storage system of claim 1 , wherein the program instructions, when executed, cause the storage system to carry out the step of:
detecting that the disruption in data communications with one or more of the other hardware-based storage systems or cloud-based storage systems has been repaired; and
making the dataset on the storage system accessible for management and dataset operations.
4. The storage system of claim 1 , wherein the program instructions, when executed, cause the storage system to carry out the step of:
identifying a target storage system for asynchronously receiving the dataset, wherein the target storage system is not one of the hardware-based storage systems or the cloud-based storage systems across which the dataset is synchronously replicated;
identifying a portion of the dataset that is not being asynchronously replicated to the target storage system by any of the other hardware-based storage systems or the cloud-based storage systems; and
asynchronously replicating, to the target storage system, the portion of the dataset that is not being asynchronously replicated to the target storage system by any of the other hardware-based storage systems or the cloud-based storage systems, wherein two or more storage systems of the hardware-based storage systems or the cloud-based storage systems collectively replicate the entire dataset to the target storage system.
5. The storage system of claim 1 , wherein the storage system is a cloud-based storage system, and wherein the program instructions, when executed, cause the storage system to carry out the step of:
receiving, by the storage system, a request to write data to the storage system;
storing, in solid-state storage of the cloud-based storage system, the data; and
storing, in object-storage of the cloud-based storage system, the data.
6. The storage system of claim 5 , wherein storing, in solid-state storage of the cloud-based storage system, the data further comprises storing, in local storage of one or more cloud computing instances, the data.
7. The storage system of claim 5 , wherein storing, in object-storage of the cloud-based storage system, the data further comprises creating one or more equal sized objects, wherein each equal sized object includes a distinct chunk of the data.
8. A method comprising:
specifying a set of managed objects, a set of management operations, and a set of access operations to correspond to a dataset that is synchronously replicated across hardware-based storage systems and cloud-based storage systems;
detecting a disruption in data communications with one or more of the other hardware-based storage systems or cloud-based storage systems;
determining whether the particular storage system should continue to synchronously replicate the dataset;
responsive to determining that the particular storage system should continue to synchronously replicate the dataset, keeping the dataset on the particular storage system accessible for management and dataset operations; and
responsive to determining that the particular storage system should not continue to synchronously replicate the dataset, making the dataset on the particular storage system inaccessible for management and dataset operations.
9. The method of claim 8 , further comprising:
receiving a request to read a portion of the dataset; and
processing the request to read the portion of the dataset locally.
10. The method of claim 8 , further comprising:
detecting that the disruption in data communications with one or more of the other hardware-based storage systems or cloud-based storage systems has been repaired; and
making the dataset on the storage system accessible for management and dataset operations.
11. The method of claim 8 , further comprising:
identifying a target storage system for asynchronously receiving the dataset, wherein the target storage system is not one of the hardware-based storage systems or the cloud-based storage systems across which the dataset is synchronously replicated;
identifying a portion of the dataset that is not being asynchronously replicated to the target storage system by any of the other hardware-based storage systems or the cloud-based storage systems; and
asynchronously replicating, to the target storage system, the portion of the dataset that is not being asynchronously replicated to the target storage system by any of the other hardware-based storage systems or the cloud-based storage systems, wherein two or more storage systems of the hardware-based storage systems or the cloud-based storage systems collectively replicate the entire dataset to the target storage system.
12. The method of claim 8 , wherein the storage system is a cloud-based storage system, and wherein the method further comprises:
receiving, by the storage system, a request to write data to the storage system;
storing, in solid-state storage of the cloud-based storage system, the data; and
storing, in object-storage of the cloud-based storage system, the data.
13. The method of claim 12 , wherein storing, in solid-state storage of the cloud-based storage system, the data further comprises storing, in local storage of one or more cloud computing instances, the data.
14. The method of claim 12 , wherein storing, in object-storage of the cloud-based storage system, the data further comprises creating one or more equal sized objects, wherein each equal sized object includes a distinct chunk of the data.
15. An apparatus comprising a computer processor, a computer memory operatively coupled to the computer processor, the computer memory having disposed within it computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of:
specifying a set of managed objects, a set of management operations, and a set of access operations to correspond to a dataset that is synchronously replicated across hardware-based storage systems and cloud-based storage systems;
detecting a disruption in data communications with one or more of the other hardware-based storage systems or cloud-based storage systems;
determining whether the particular storage system should continue to synchronously replicate the dataset;
responsive to determining that the particular storage system should continue to synchronously replicate the dataset, keeping the dataset on the particular storage system accessible for management and dataset operations; and
responsive to determining that the particular storage system should not continue to synchronously replicate the dataset, making the dataset on the particular storage system inaccessible for management and dataset operations.
16. The apparatus of claim 15 , further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of:
receiving a request to read a portion of the dataset; and
processing the request to read the portion of the dataset locally.
17. The apparatus of claim 15 , wherein the storage system is a cloud-based storage system, and wherein the computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of:
receiving, by the storage system, a request to write data to the storage system;
storing, in solid-state storage of the cloud-based storage system, the data; and
storing, in object-storage of the cloud-based storage system, the data.
18. The method of claim 1 wherein each hardware-based storage system and each cloud-based storage system stores a separate copy of the dataset as a proper subset of the datasets stored.
19. The method of claim 1 wherein operations to modify managed objects or the dataset performed and completed through the hardware-based storage systems and the cloud-based storage systems are reflected in subsequent management objects to query the dataset or subsequent access operations to read the dataset.
20. The method of claim 1 wherein access operations to read or modify the dataset operate equivalently through the hardware-based storage systems and the cloud-based storage systems.