Establishing a guarantee for maintaining a replication relationship between object stores during a communications outage
Establishing a guarantee for maintaining a replication relationship between object stores during a communications outage, an embodiment including identifying, by a first storage system, respective local clock values of one or more second storage systems, wherein the first storage system and the one or more second storage systems are among a plurality of storage systems replicating an object store, wherein the plurality of storage systems are configured to receive requests directed to the replicated object store; determining, by the first storage system in dependence upon the respective local clock values, respective clock differences of the one or more second storage systems relative to the first storage system; and ordering, by the first storage system, one or more updates to the replicated object store in dependence upon the respective clock differences.
1. A method comprising:
identifying, by a first storage system, respective local clock values of one or more second storage systems, wherein the first storage system and the one or more second storage systems are among a plurality of storage systems replicating a replicated object store, wherein the plurality of storage systems are configured to receive requests directed to the replicated object store;
determining, by the first storage system in dependence upon the respective local clock values, respective clock differences of the one or more second storage systems relative to the first storage system;
identifying, by the first storage system, clock uncertainty values associated with one or more local clock values for the one or more second storage systems relative to the first storage system, the clock uncertainty values including an imprecision value that accounts for a range of clock uncertainty values; and
ordering, by the first storage system, two or more updates to the replicated object store in dependence upon the respective clock differences and the clock uncertainty values.
2. The method of claim 1 , wherein the clock uncertainty value is based on a latency between sending a clock request message and receiving a clock response message.
3. The method of claim 1 further comprising:
identifying, by the first storage system, a worst-case clock uncertainty among the plurality of storage systems;
determining, in dependence upon the worst-case clock uncertainty, a clock coordination precision for the plurality of storage systems; and
indicating, by the first storage system to one or more hosts, the clock coordination precision.
4. The method of claim 3 , wherein the clock coordination precision indicates a minimum amount of time separating two object update requests directed to separate storage systems of the plurality of storage systems such that the plurality of storage systems can ensure correct ordering of the updates.
5. The method of claim 3 further comprising:
determining validity duration for the clock coordination precision; and
indicating, to one or more hosts, the validity duration.
6. The method of claim 1 further comprising:
identifying a fault detection time, wherein the fault detection time is an amount of time to detect a disruption in communications; and
indicating the fault detection time to one or more hosts.
7. The method of claim 6 further comprising:
delaying, in response to identifying that the fault detection time has expired, local processing of requests directed to the replicated object store.
8. The method of claim 6 further comprising:
sending, in response to identifying that fault detection time has expired, a message to a mediation service; and
receiving a mediation result, wherein the mediation result indicates whether local processing of requests directed to the replicated object store should be disabled on the first storage system.
9. The method of claim 6 further comprising:
initiating, in response to identifying that fault detection time has expired, a quorum protocol; and
determining, in dependence upon a quorum protocol result, that the first storage system is a member of a quorum.
10. The method of claim 6 further comprising:
updating a clock coordination precision in response to identifying that fault detection time has expired, including increasing a value of the clock coordination precision; and
indicating the updated clock coordination precision to the one or more hosts.
11. The method of claim 1 , wherein the replicated object store is a bidirectionally-replicated multi-site bucket.
12. The method of claim 1 , wherein at least two storage systems of the plurality of storage systems are in different cloud service provider regions.
13. An apparatus comprising:
a memory; and
a processor, operatively coupled to the memory, configured to:
identify, by a first storage system, respective local clock values of one or more second storage systems, wherein the first storage system and the one or more second storage systems are among a plurality of storage systems replicating a replicated object store, wherein the plurality of storage systems are configured to receive requests directed to the replicated object store;
determine, by the first storage system in dependence upon the respective local clock values, respective clock differences of the one or more second storage systems relative to the first storage system;
identify, by the first storage system, clock uncertainty values associated with one or more local clock values for the one or more second storage systems relative to the first storage system, the clock uncertainty values including an imprecision value that accounts for a range of clock uncertainty values; and
order, by the first storage system, two or more updates to the replicated object store in dependence upon the respective clock differences and the clock uncertainty values.
14. The apparatus of claim 13 , wherein the clock uncertainty value is based on a latency between sending a clock request message and receiving a clock response message.
15. The apparatus of claim 13 , the processor further configured to:
identify, by the first storage system, a worst-case clock uncertainty among the plurality of storage systems;
determine, in dependence upon the worst-case clock uncertainty, a clock coordination precision for the plurality of storage systems; and
indicate, by the first storage system to one or more hosts, the clock coordination precision.
16. The apparatus of claim 15 , wherein the clock coordination precision indicates a minimum amount of time separating two object update requests directed to separate storage systems of the plurality of storage systems such that the plurality of storage systems can ensure correct ordering of the updates.
17. The apparatus of claim 15 , the processor further configured to:
determine validity duration for the clock coordination precision; and
indicate indicating, to one or more hosts, the validity duration.
18. A non-transitory computer readable storage medium storing instructions which, when executed, cause a processor to:
identify, by a first storage system, respective local clock values of one or more second storage systems, wherein the first storage system and the one or more second storage systems are among a plurality of storage systems replicating a replicated object store, wherein the plurality of storage systems are configured to receive requests directed to the replicated object store;
determine, by the first storage system in dependence upon the respective local clock values, respective clock differences of the one or more second storage systems relative to the first storage system;
identify, by the first storage system, clock uncertainty values associated with one or more local clock values for the one or more second storage systems relative to the first storage system, the clock uncertainty values including an imprecision value that accounts for a range of clock uncertainty values; and
order, by the first storage system, two or more updates to the replicated object store in dependence upon the respective clock differences and the clock uncertainty values.