Cluster failover to avoid network partitioning
During a synchronization technique, states of a primary cluster in the computer system with multiple primary controllers that provide controllers for access points and a backup cluster in the computer system with multiple backup controllers that independently provide controllers for the access points may be dynamically synchronized. In particular the primary cluster may receive configuration requests with configuration information for the access points on an input node of the primary cluster. In response, the primary cluster may store the configuration requests in a replay queue in the computer system. Then, the primary cluster may playback the configuration requests in the replay queue for the backup cluster to synchronize the states of the primary cluster and the backup cluster. For example, the configuration requests may be played back within a time interval associated with a service level agreement of a service provider of a service for the access points.
1. A computer system, comprising:
a primary cluster with multiple primary controllers and a replay queue, wherein the primary cluster is configured to communicate with access points and a backup cluster; and
the backup cluster with multiple backup controllers coupled to the primary cluster, wherein the backup cluster is configured to communicate with the access points and the primary cluster, wherein the primary controllers and the backup controllers are independently configured to implement controllers for the access points, and wherein the primary cluster is configured to:
receive configuration requests with configuration information for the access points on an input node of the primary cluster;
store the configuration requests in the replay queue; and
playback the configuration requests in the replay queue for the backup cluster to synchronize states of the primary cluster and the backup cluster, wherein the configuration requests are played back in order for the backup cluster.
2. The computer system of claim 1 , wherein the access points are divided into zones and a given zone is associated with a given primary controller and a given backup controller.
3. The computer system of claim 2 , wherein the zones are based on one or more of: geographic locations of the access points, and software versions of the access points.
4. The computer system of claim 2 , wherein the computer system is configured to provide network addresses of the given primary controller and the given backup controller to an output node of the computer system for a subset of the access points in the given zone.
5. The computer system of claim 2 , wherein, when the given primary controller fails, the primary cluster is configured to:
receive requests associated with a subset of the access points in the given zone on the input node of the primary cluster; and
in response to the requests, provide, using at least another primary controller, failover coverage for the subset of the access points.
6. The computer system of claim 5 , wherein the requests are independently received from the subset of the access points.
7. The computer system of claim 5 , wherein the primary cluster is configured to select the other primary controller based on loading of the primary controllers.
8. The computer system of claim 5 , wherein, when the other primary controller subsequently fails, the backup cluster is configured to:
receive additional requests associated with the subset of the access points on an input node of the backup cluster; and
in response to the additional requests, provide, using at least the given backup controller, failover coverage for the subset of the access points.
9. The computer system of claim 8 , wherein the additional requests are independently received from the subset of the access points.
10. The computer system of claim 1 , wherein the configuration requests are played back within a time interval.
11. The computer system of claim 10 , wherein the time interval is associated with a service level agreement of a service provider of a service for the access points.
12. The computer system of claim 1 , wherein, when a link between the primary cluster and the backup cluster fails, the primary cluster is configured to:
store additional configuration requests that are subsequently received in the replay queue; and
playback the additional configuration requests in the replay queue for the backup cluster when the link is restored; and
wherein the additional configuration requests are played back in order for the backup cluster.
13. A non-transitory computer-readable storage medium for use in conjunction with a computer system that includes a primary cluster with multiple primary controllers and a backup cluster with multiple backup controllers, and the primary controllers and the backup controllers are independently configured to implement controllers for access points, the computer-readable storage medium storing program instructions that, when executed by the computer system, causes the computer system to perform one or more operations comprising:
receiving configuration requests for the primary cluster with configuration information for the access points;
storing the configuration requests in a replay queue; and
playing back the configuration requests in the replay queue for the backup cluster to synchronize states of the primary cluster and the backup cluster, wherein the configuration requests are played back in order for the backup cluster.
14. The computer-readable storage medium of claim 13 , wherein the access points are divided into zones and a given zone is associated with a given primary controller and a given backup controller.
15. The computer-readable storage medium of claim 14 , wherein, when the given primary controller fails, the one or more operations comprise:
receiving requests associated with a subset of the access points in the given zone; and
in response to the requests, providing, using at least another primary controller, failover coverage for the subset of the access points.
16. The computer-readable storage medium of claim 15 , wherein, when the other primary controller subsequently fails, the one or more operations comprise:
receiving additional requests associated with the subset of the access points; and
in response to the additional requests, providing, using at least the given backup controller, failover coverage for the subset of the access points.
17. The computer-readable storage medium of claim 13 , wherein, when a link between the primary cluster and the backup cluster fails, the one or more operations comprise:
storing additional configuration requests that are subsequently received in the replay queue; and
playing back the additional configuration requests in the replay queue for the backup cluster when the link is restored; and
wherein the additional configuration requests are played back in order for the backup cluster.
18. A method for synchronizing states of a primary cluster with multiple primary controllers and a backup cluster with multiple backup controllers, wherein the primary controllers and the backup controllers independently implement controllers for access points, comprising:
by a computer system:
receiving configuration requests for the primary cluster with configuration information for the access points;
storing the configuration requests in a replay queue; and
playing back the configuration requests in the replay queue for the backup cluster to synchronize states of the primary cluster and the backup cluster, wherein the configuration requests are played back in order for the backup cluster.
19. The method of claim 18 , wherein the access points are divided into zones and a given zone is associated with a given primary controller and a given backup controller; and
wherein, when the given primary controller fails, the method comprises:
receiving requests associated with a subset of the access points in the given zone; and
in response to the requests, providing, using at least another primary controller, failover coverage for the subset of the access points.
20. The method of claim 19 , wherein, when the other primary controller subsequently fails, the method comprises:
receiving additional requests associated with the subset of the access points; and
in response to the additional requests, providing, using at least the given backup controller, failover coverage for the subset of the access points.