High availability and disaster recovery for replicated object stores
High availability and disaster recovery for replicated object stores is disclosed. An embodiment includes receiving, by a first storage system of a plurality of storage systems symmetrically replicating objects of a bucket, a request to establish immutable content for the bucket; indicating, by the first storage system to a second storage system of the plurality of storage systems, the request to establish immutable content, wherein the second storage system establishes an ordering for conflicting requests of different storage systems to establish immutable content for the bucket; and processing, by the first storage system, the request based on ordering information received from the second storage system.
1 . A method comprising:
receiving, by a first storage system of a plurality of storage systems symmetrically replicating objects of a bucket, a request to establish immutable content for the bucket;
indicating, by the first storage system to a second storage system of the plurality of storage systems, the request to establish immutable content, wherein the second storage system establishes an ordering for conflicting requests with a same object name, the conflicting requests being of different storage systems to establish immutable content for the bucket; and
processing, by the first storage system, the request based on ordering information received from the second storage system.
2 . The method of claim 1 , wherein the request is a first operation to establish first immutable content for an object using an object name, and wherein a conflicting request is a second operation to establish second immutable content for another object using the object name, and wherein the ordering establishes that a first object replaces a second object for the object name in the bucket.
3 . The method of claim 2 , wherein the second object is retained as a non-current version of the first object for the object name in the bucket.
4 . The method of claim 1 , wherein the request is a first operation to establish first immutable content for a first object as a copy of a second object associated with an object name; wherein a conflicting operation is a second operation to establish third immutable content for a third object using the object name to replace the second object that includes second immutable content; and wherein the ordering determines whether to copy either the second immutable content or the third immutable content.
5 . The method of claim 1 , wherein two or more storage systems of the plurality of storage systems synchronously replicate the bucket such that a request for an operation to a first storage system of the two or more storage systems is not considered completed until a result of the operation has been consistently applied to each of the two or more storage systems.
6 . The method of claim 1 further comprising:
detecting, by the first storage system, a fault in communication with at least one other storage system of the plurality of storage systems; and
determining, by the first storage system, to continue operating with the at least one other storage system as a removed storage system that is at least temporarily removed from symmetrically replicating objects of the bucket.
7 . The method of claim 6 , wherein the first storage system determines that it should continue operating based on winning a request for mediation from a mediator.
8 . The method of claim 6 , wherein the first storage system is a member of a subset of the plurality of storage systems that determines that it should continue operating based on determining that the subset comprises a quorum of storage systems symmetrically replicating objects of the bucket.
9 . The method of claim 6 , wherein, upon the removed storage system resuming communication the first storage system, the bucket is resynchronized to the removed storage system, and the removed storage system is rejoined among the plurality of storage systems symmetrically replicating objects of the bucket.
10 . The method of claim 9 , wherein resynchronizing comprises copying missing immutable content of objects of the bucket to the removed storage system and establishing content and names of missing objects, content, identities, and orderings of missing versions of objects.
11 . The method of claim 1 , wherein the plurality of storage systems are dispersed across two or more regions, wherein a first mode of replication is used for replication within a region, and wherein a second mode of replication is used for replication across regions.
12 . The method of claim 11 , wherein the first mode of replication is synchronous replication.
13 . The method of claim 12 , wherein at least two storage systems within a region are located in different availability zones of the region.
14 . The method of claim 11 , wherein the second mode of replication is non-synchronous replication using an eventual consistency model.
15 . The method of claim 1 , wherein the plurality of storage systems directionally and non-synchronously replicate the bucket to one or more additional storage systems.
16 . An apparatus comprising
a memory; and
a processor, operatively coupled to the memory, configured to:
receive, by a first storage system of a plurality of storage systems symmetrically replicating objects of a bucket, a request to establish immutable content for the bucket;
indicate, by the first storage system to a second storage system of the plurality of storage systems, the request to establish immutable content, wherein the second storage system establishes an ordering for conflicting requests with a same object name, the conflicting requests being of different storage systems to establish immutable content for the bucket; and
process, by the first storage system, the request based on ordering information received from the second storage system.
17 . The apparatus of claim 16 , the processor further configured to:
detect, by the first storage system, a fault in communication with at least one other storage system of the plurality of storage systems; and
determine, by the first storage system, to continue operating with the at least one other storage system as a removed a storage system that is at least temporarily removed from symmetrically replicating objects of the bucket.
18 . The apparatus of claim 17 , wherein the first storage system determines that it should continue operating based on winning a request for mediation from a mediator.
19 . The apparatus of claim 17 , wherein the first storage system is a member of a subset of the plurality of storage systems that determines that it should continue operating based on determining that the subset comprises a quorum of storage systems symmetrically replicating objects of the bucket.
20 . A non-transitory computer readable storage medium storing instructions, which when executed, cause a processor to:
receive, by a first storage system of a plurality of storage systems symmetrically replicating objects of a bucket, a request to establish immutable content for the bucket;
indicate, by the first storage system to a second storage system of the plurality of storage systems, the request to establish immutable content, wherein the second storage system establishes an ordering for conflicting requests with a same object name, the conflicting requests being of different storage systems to establish immutable content for the bucket; and
process, by the first storage system, the request based on ordering information received from the second storage system.