Failover and recovery for replicated data instances
Replicated instances in a database environment provide for automatic failover and recovery. A monitoring component can periodically communicate with a primary and a secondary replica for an instance, with each capable of residing in a separate data zone or geographic location to provide a level of reliability and availability. A database running on the primary instance can have information synchronously replicated to the secondary replica at a block level, such that the primary and secondary replicas are in sync. In the event that the monitoring component is not able to communicate with one of the replicas, the monitoring component can attempt to determine whether those replicas can communicate with each other, as well as whether the replicas have the same data generation version. Depending on the state information, the monitoring component can automatically perform a recovery operation, such as to failover to the secondary replica or perform secondary replica recovery.
1. A system, comprising:
a plurality of computing nodes, respectively comprising at least one processor and a memory that together implement a control plane for a data store;
the control plane, configured to:
obtain, by a monitoring component, data generation information for each of a primary instance replica and a secondary instance replica, wherein the primary instance replica and the secondary instance replica are located in different data zones;
detect, by the monitoring component, a loss of communication in at least one direction between the monitoring component and the primary instance replica or the secondary instance replica, or in at least one direction between the primary instance replica and the secondary instance replica; and
in response to the detection of the loss of communication in at least one direction, perform a particular type of failover operation or recovery process, wherein the particular type of failover operation or recovery process is determined based at least in part on a nature of the detected loss of communication and the respective data generation information for the primary instance replica and the secondary instance replica.
2. The system of claim 1 ,
wherein to detect the loss of communication in at least one direction between the monitoring component and the primary instance replica or the secondary instance replica, the control plane is configured to determine that the primary instance replica or the secondary instance replica failed to respond to a request for status from the monitoring component;
wherein to detect the loss of communication in at least one direction between the primary instance replica and the secondary instance replica, the control plane is configured to receive an indication from the primary instance replica of the loss of communication between the primary instance replica and the secondary instance replica or receive an indication from the secondary instance replica of the loss of communication between the secondary instance replica and the primary instance replica.
3. The system of claim 1 ,
wherein the detected loss of communication is between the monitoring component and the primary instance replica;
wherein a further loss of communication is detected between the secondary instance replica and the primary instance replica;
wherein the control plane is further configured to determine that the data generation information for the primary instance replica and the secondary instance replica match; and
wherein the particular type of failover operation or recovery process is a failover operation to promote the secondary instance replica to operate as a new primary instance replica.
4. The system of claim 1 , wherein the detected loss of communication is between the monitoring component and the secondary instance replica, wherein a further loss of communication is detected between the primary instance replica and the secondary instance replica, and wherein the particular type of failover operation or recovery process is a secondary instance recovery process.
5. The system of claim 1 , wherein the control plane is further configured to:
store by the monitoring component, recovery information comprising the data generation information for each of the primary instance replica and the secondary instance replica and the detected loss of communication in a control plane data store; and
access the recovery information in the control plane data store to generate a workflow to be executed, wherein the particular failover operation or recovery process is performed as part of executing the generated workflow.
6. The system of claim 1 , wherein the monitoring component is located in another data zone different than either the primary instance replica or the secondary instance replica.
7. The system of claim 1 , wherein the monitoring component is one of a plurality of monitoring components implemented as part of the control plane that monitors a plurality of primary and secondary instance replicas including the primary instance replica and the secondary instance replica, wherein prior to the obtainment of the data generation information for the primary instance replica and the secondary instance replica, the monitoring component obtained a lease to monitor the primary instance replica and the secondary instance replica that excludes other ones of the plurality of monitoring components from monitoring the primary instance replica and the secondary instance replica.
8. A method, comprising:
performing, by one or more computers,
obtaining, by a monitoring component, data generation information for each of a primary instance replica and a secondary instance replica, wherein the primary instance replica and the secondary instance replica are located in different data zones;
detecting, by the monitoring component, a loss of communication in at least one direction between the monitoring component and the primary instance replica or the secondary instance replica, or in at least one direction between the primary instance replica and the secondary instance replica; and
in response to detecting the loss of communication in at least one direction, performing a particular type of failover operation or recovery process, wherein the particular type of failover operation or recovery process is determined based at least in part on a nature of the detected loss of communication and the respective data generation information for the primary instance replica and the secondary instance replica.
9. The method of claim 8 ,
wherein detecting the loss of communication in at least one direction between the monitoring component and the primary instance replica or the secondary instance replica comprises determining that the primary instance replica or the secondary instance replica failed to respond to a request for status from the monitoring component;
wherein detecting the loss of communication in at least one direction between the primary instance replica and the secondary instance replica comprises receiving an indication from the primary instance replica of the loss of communication between the primary instance replica and the secondary instance replica or receiving an indication from the secondary instance replica of the loss of communication between the secondary instance replica and the primary instance replica.
10. The method of claim 8 ,
wherein the detected loss of communication is between the monitoring component and the primary instance replica;
wherein a further loss of communication is detected between the secondary instance replica and the primary instance replica;
wherein the method further comprises determining that the data generation information for the primary instance replica and the secondary instance replica match; and
wherein the particular type of failover operation or recovery process is a failover operation to promote the secondary instance replica to operate as a new primary instance replica.
11. The method of claim 8 , wherein the detected loss of communication is between the monitoring component and the secondary instance replica, wherein a further loss of communication is detected between the primary instance replica and the secondary instance replica, and wherein the particular type of failover operation or recovery process is a secondary instance recovery process.
12. The method of claim 8 , further comprising:
storing, by the monitoring component, recovery information comprising the data generation information for each of the primary instance replica and the secondary instance replica and the detected loss of communication in a control plane data store; and
accessing the recovery information in the control plane data store to generate a workflow to be executed, wherein the particular failover operation or recovery process is performed as part of executing the generated workflow.
13. The method of claim 8 , wherein the monitoring component is located in another data zone different than either the primary instance replica or the secondary instance replica.
14. The method of claim 8 , wherein the monitoring component is one of a plurality of monitoring components implemented as part of a control plane that monitors a plurality of primary and secondary instance replicas including the primary instance replica and the secondary instance replica, wherein prior to obtaining the data generation information for the primary instance replica and the secondary instance replica, the monitoring component obtained a lease to monitor the primary instance replica and the secondary instance replica that excludes other ones of the plurality of monitoring components from monitoring the primary instance replica and the secondary instance replica.
15. A non-transitory, computer-readable storage medium, comprising program instructions that when executed by the one or more computing devices cause the one or more computing devices to implement:
obtaining, by a monitoring component, data generation information for each of a primary instance replica and a secondary instance replica, wherein the primary instance replica and the secondary instance replica are located in different data zones;
detecting, by the monitoring component, a loss of communication in at least one direction between the monitoring component and the primary instance replica or the secondary instance replica, or in at least one direction between the primary instance replica and the secondary instance replica; and
in response to detecting the loss of communication in at least one direction, performing a particular type of failover operation or recovery process, wherein the particular type of failover operation or recovery process is determined based at least in part on a nature of the detected loss of communication and the respective data generation information for the primary instance replica and the secondary instance replica.
16. The non-transitory, computer-readable storage medium of claim 15 ,
wherein, in detecting the loss of communication in at least one direction between the monitoring component and the primary instance replica or the secondary instance replica, the program instructions cause the one or more computing devices to implement determining that the primary instance replica or the secondary instance replica failed to respond to a request for status from the monitoring component;
wherein, in detecting the loss of communication in at least one direction between the primary instance replica and the secondary instance replica, the program instructions cause the one or more computing devices to implement receiving an indication from the primary instance replica of the loss of communication between the primary instance replica and the secondary instance replica or receiving an indication from the secondary instance replica of the loss of communication between the secondary instance replica and the primary instance replica.
17. The non-transitory, computer-readable storage medium of claim 15 ,
wherein the detected loss of communication is between the monitoring component and the primary instance replica;
wherein a further loss of communication is detected between the secondary instance replica and the primary instance replica;
wherein the program instructions cause the one or more computing devices to further implement determining that the data generation information for the primary instance replica and the secondary instance replica match; and
wherein the particular type of failover operation or recovery process is a failover operation to promote the secondary instance replica to operate as a new primary instance replica.
18. The non-transitory, computer-readable storage medium of claim 15 , wherein the detected loss of communication is between the monitoring component and the secondary instance replica, wherein a further loss of communication is detected between the primary instance replica and the secondary instance replica, and wherein the particular type of failover operation or recovery process is a secondary instance recovery process.
19. The non-transitory, computer-readable storage medium of claim 15 , wherein the program instructions cause the one or more computing devices to implement:
storing, by the monitoring component, recovery information comprising the data generation information for each of the primary instance replica and the secondary instance replica and the detected loss of communication in a control plane data store; and
accessing the recovery information in the control plane data store to generate a workflow to be executed, wherein the particular failover operation or recovery process is performed as part of executing the generated workflow.
20. The non-transitory, computer-readable storage medium of claim 15 , wherein the monitoring component is located in another data zone different than either the primary instance replica or the secondary instance replica.