Use of backup heartbeat signal to mitigate split-brain conditions
Approaches to using a backup heartbeat interface to detect and mitigate high availability (HA) split-brain conditions are described. Heartbeat packets from the primary appliance received via a regular heartbeat interface of the backup appliance and monitored the backup appliance to determine whether the connection with the primary appliance via the regular heartbeat interface is stable. A backup heartbeat interface on the backup appliance is enabled in response to a determination that the primary appliance is not operating properly resulting from analyzing at least a status of the heartbeat packets from the primary appliance. A backup heartbeat packet is transmitted from the backup appliance to at least the regular heartbeat interface of the primary appliance in response to a determination that the primary appliance is not operating properly. Services are provided with the backup appliance to replace services previously provided by the primary appliance. Transmission of the backup heartbeat packets from the backup appliance continues while providing services with the backup appliance.
1 . A method for managing heartbeat signals in a high availability (HA) cluster having at least a primary appliance and a backup appliance, the method comprising:
monitoring heartbeat packets from the primary appliance received via a regular heartbeat interface of the backup appliance with the backup appliance to determine whether the connection with the primary appliance via the regular heartbeat interface is stable;
enabling a backup heartbeat interface on the backup appliance in response to a determination that the primary appliance is not operating properly resulting from analyzing at least a status of the heartbeat packets from the primary appliance;
transmitting a backup heartbeat packet from the backup appliance to at least the regular heartbeat interface of the primary appliance in response to a determination that the primary appliance is not operating properly;
providing services with the backup appliance to replace services previously provided by the primary appliance; and
continuing transmission of backup heartbeat packets from the backup appliance while providing services with the backup appliance.
2 . The method of claim 1 further comprising:
detecting a return of a regular heartbeat signal from the primary appliance with the backup appliance; and
transmitting a backup heartbeat packet with an indication to discontinue use of the backup heartbeat packets.
3 . The method of claim 2 , wherein transmitting a backup heartbeat packet with an indication to discontinue use of the backup heartbeat packets causes appliances of the HA cluster to disable local backup heartbeat interfaces.
4 . The method of claim 2 further comprising providing the services with the primary appliance after use of the backup heartbeat packets has been discontinued.
5 . The method of claim 1 , wherein the primary appliance and the backup appliance comprise network security gateway devices coupled as an HA pair.
6 . The method of claim 1 , wherein the primary appliance, the backup appliance and one or secondary appliances comprise network security gateway devices coupled as an HA cluster.
7 . The method of claim 1 , wherein transmitting a backup heartbeat packet from the backup appliance to regular heartbeat interface of the primary appliance in response to a determination that the primary appliance is not operating properly further comprises also transmitting the backup heartbeat packet to regular heartbeat interfaces of one or more secondary appliances of an HA cluster.
8 . A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to:
monitor heartbeat packets from the primary appliance received via a regular heartbeat interface of the backup appliance with the backup appliance to determine whether the connection with the primary appliance via the regular heartbeat interface is stable;
enable a backup heartbeat interface on the backup appliance in response to a determination that the primary appliance is not operating properly resulting from analyzing at least a status of the heartbeat packets from the primary appliance;
transmit a backup heartbeat packet from the backup appliance to at least the regular heartbeat interface of the primary appliance in response to a determination that the primary appliance is not operating properly;
provide services with the backup appliance to replace services previously provided by the primary appliance; and
continue transmission of backup heartbeat packets from the backup appliance while providing services with the backup appliance.
9 . The non-transitory computer-readable storage medium of claim 8 further comprising instructions that, when executed by the one or more processors, cause the one or more processors to:
detect a return of a regular heartbeat signal from the primary appliance with the backup appliance; and
transmit a backup heartbeat packet with an indication to discontinue use of the backup heartbeat packets.
10 . The non-transitory computer-readable storage medium of claim 9 , wherein transmitting a backup heartbeat packet with an indication to discontinue use of the backup heartbeat packets causes appliances of the HA cluster to disable local backup heartbeat interfaces.
11 . The non-transitory computer-readable storage medium of claim 9 further comprising instructions that, when executed by the one or more processors, cause the one or more processors to provide the services with the primary appliance after use of the backup heartbeat packets has been discontinued.
12 . The non-transitory computer-readable storage medium of claim 8 , wherein the primary appliance and the backup appliance comprise network security gateway devices coupled as an HA pair.
13 . The non-transitory computer-readable storage medium of claim 8 , wherein the primary appliance, the backup appliance and one or secondary appliances comprise network security gateway devices coupled as an HA cluster.
14 . The non-transitory computer-readable storage medium of claim 8 , wherein transmitting a backup heartbeat packet from the backup appliance to regular heartbeat interface of the primary appliance in response to a determination that the primary appliance is not operating properly further comprises also transmitting the backup heartbeat packet to regular heartbeat interfaces of one or more secondary appliances of an HA cluster.
15 . A system comprising:
a memory system having a plurality of interconnected memory devices;
one or more hardware processors coupled with the memory system, the one or more hardware processors configurable to:
monitor heartbeat packets from the primary appliance received via a regular heartbeat interface of the backup appliance with the backup appliance to determine whether the connection with the primary appliance via the regular heartbeat interface is stable;
enable a backup heartbeat interface on the backup appliance in response to a determination that the primary appliance is not operating properly resulting from analyzing at least a status of the heartbeat packets from the primary appliance;
transmit a backup heartbeat packet from the backup appliance to at least the regular heartbeat interface of the primary appliance in response to a determination that the primary appliance is not operating properly;
provide services with the backup appliance to replace services previously provided by the primary appliance; and
continue transmission of backup heartbeat packets from the backup appliance while providing services with the backup appliance.
16 . The system of claim 15 wherein the one or more hardware processors are further configurable to:
detect a return of a regular heartbeat signal from the primary appliance with the backup appliance; and
transmit a backup heartbeat packet with an indication to discontinue use of the backup heartbeat packets.
17 . The system of claim 16 , wherein transmitting a backup heartbeat packet with an indication to discontinue use of the backup heartbeat packets causes appliances of the HA cluster to disable local backup heartbeat interfaces.
18 . The system of claim 16 wherein the one or more processors are further configurable to provide the services with the primary appliance after use of the backup heartbeat packets has been discontinued.
19 . The system of claim 15 , wherein the primary appliance and the backup appliance comprise network security gateway devices coupled as an HA pair.
20 . The system of claim 15 , wherein the primary appliance, the backup appliance and one or secondary appliances comprise network security gateway devices coupled as an HA cluster.