Methods and systems for negotiating a primary bias state in a distributed storage system
Systems and methods include negotiating a primary bias state for primary and secondary storage sites when a mediator is temporarily unavailable for a multi-site distributed storage system. In one example, a computer-implemented method comprises detecting, with the primary storage site having a primary storage cluster, a temporary loss of connectivity to a mediator or a failure of the mediator. The computer-implemented method includes negotiating the primary bias state and setting the primary bias state on a secondary storage cluster of the secondary storage site when the secondary storage cluster detects a temporary loss of connectivity to the mediator, determining whether the primary storage cluster receives a confirmation of the secondary storage cluster setting the primary bias state, and setting the primary bias state on the primary storage cluster when the primary storage cluster receives the confirmation.
1. A computer-implemented method executed by one or more processing resources, comprising:
detecting, with a primary storage site of a multi-site distributed storage system having a primary storage cluster, a temporary loss of connectivity to a mediator located remotely from the primary storage site;
negotiating a primary bias state and setting the primary bias state on a secondary storage cluster of the secondary storage site in response to the secondary storage cluster also independently detecting a temporary loss of connectivity to the mediator;
determining whether the primary storage cluster receives a confirmation of the secondary storage cluster setting the primary bias state; and
setting the primary bias state on the primary storage cluster when the primary storage cluster receives the confirmation.
2. The computer-implemented method of claim 1 , further comprising: waiting for the confirmation of the secondary storage cluster setting a primary bias state.
3. The computer-implemented method of claim 1 , further comprising:
receiving a rejection of the primary bias state if the mediator is communicatively reachable from the secondary storage cluster.
4. The computer-implemented method of claim 1 , further comprising:
in response to a primary bias state being set in the primary storage cluster and the secondary storage cluster, serving, with the primary storage cluster, input/output (I/O) operations.
5. The computer-implemented method of claim 1 , further comprising:
detect, with at least one of the primary storage cluster and the secondary storage cluster, resumption of a connection to the mediator; and
reestablish, with the primary storage cluster, the secondary storage cluster, and the mediator, a three way quorum including updating a relationship state to the mediator and clearing a primary bias state on the primary storage cluster.
6. The computer-implemented method of claim 1 , wherein the primary storage cluster is selected as an authority based on the primary storage cluster initially being assigned a leader role for serving I/O operations and subsequently setting the primary bias state in a configuration database of the primary storage cluster and setting the primary bias state in a configuration database of the secondary storage cluster.
7. The computer-implemented method of claim 1 , wherein for a mediator failure, negotiating and setting the primary bias state in a configuration database of the primary storage cluster and in a configuration database of the secondary storage cluster before a failure of an intercluster link between the primary storage site and the secondary storage site.
8. The computer-implemented method of claim 1 , wherein for a given mediator failure, negotiating and setting the primary bias state in a configuration database of the primary storage cluster and in a configuration database of the secondary storage cluster before a failure at the secondary storage cluster.
9. A multi-site distributed storage system comprising:
one or more processing resources; and
a non-transitory computer-readable medium coupled to the one or more processing resources, having stored therein instructions, which when executed by the one or more processing resources cause the one or more processing resources to:
detect, with a primary storage site having a primary storage cluster, a temporary loss of connectivity to a mediator located remotely from the primary storage site;
negotiate a primary bias state and set the primary bias state on a secondary storage cluster of a secondary storage site in response to the secondary storage cluster also independently detecting a temporary loss of connectivity to the mediator;
determine whether the primary storage cluster receives a confirmation of the secondary storage cluster setting the primary bias state; and
set the primary bias state on the primary storage cluster when the primary storage cluster receives the confirmation.
10. The multi-site distributed storage system of claim 9 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
negotiate the primary bias state and set the primary bias state on a third storage cluster of a third storage site when the third storage cluster detects a temporary loss of connectivity to the mediator.
11. The multi-site distributed storage system of claim 10 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
determine whether the primary storage cluster receives a confirmation of the third storage cluster setting the primary bias state.
12. The multi-site distributed storage system of claim 11 , wherein the primary and secondary storage sites are located in a first region and the third storage site is located in a second region for cloud resident datasets.
13. The multi-site distributed storage system of claim 9 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
detect resumption of a connection to the mediator; and
reestablish, with the primary storage cluster, the secondary storage cluster, and the mediator, a three way quorum including updating a relationship state to the mediator and clearing a primary bias state on the primary storage cluster.
14. The multi-site distributed storage system of claim 9 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
in response to a temporary mediator failure, negotiate and set the primary bias state in a configuration database of the primary storage cluster and in a configuration database of the secondary storage cluster before a temporary failure at the secondary storage cluster and before a temporary failure of an intercluster link between the primary storage site and the secondary storage site.
15. A non-transitory computer-readable storage medium embodying a set of instructions, which when executed by one or more processing resources of a primary storage site of a multi-site distributed storage system cause the one or more processing resources to:
detect, with a primary storage site having a primary storage cluster, a temporary loss of connectivity to a mediator located remotely from the primary storage site;
negotiate a primary bias state and set the primary bias state on a secondary storage cluster in response to the secondary storage cluster also independently detecting a temporary loss of connectivity to the mediator;
determine whether the primary storage cluster receives a confirmation of the secondary storage cluster setting the primary bias state; and
set the primary bias state on the primary storage cluster when the primary storage cluster receives the confirmation.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
wait for the confirmation of the secondary storage cluster setting a primary bias state.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
receive a rejection of the primary bias state if the mediator is communicatively reachable from the secondary storage cluster.
18. The non-transitory computer-readable storage medium of claim 15 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
in response to a primary bias state being set in the primary storage cluster and the secondary storage cluster, serve, with the primary storage cluster, input/output (I/O) operations.
19. The non-transitory computer-readable storage medium of claim 15 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
detect resumption of a connection to the mediator; and
reestablish, with the primary storage cluster, the secondary storage cluster, and the mediator, a three way quorum including updating a relationship state to the mediator and clearing a primary bias state on the primary storage cluster.
20. The non-transitory computer-readable storage medium of claim 15 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
in response to a temporary mediator failure, negotiate and set the primary bias state in a configuration database of the primary storage cluster and in a configuration database of the secondary storage cluster before a failure of an intercluster link between the primary storage site and the secondary storage site.