Failure detection and fencing in a computing system
View Patent ↗A method, computer program product, and system for detecting and fencing off a failed entity instance so that failover time in the computing system is reduced. Upon detection of a failed entity, for example a failed process, a signal handler invokes an operating system interface to fence off the failed instance from modifying its persistent state, so that the entity may be re-instantiated prior to termination of the failed instance. This approach reduces failover time and eliminates split-brain problems without compromising access to the failed instance to obtain diagnostic information, core dumps, and the like.
1. A method for detecting and fencing off a failed entity instance comprising:
detecting a failure occurring during execution of a first instance of a software entity, wherein the software entity is an application or process and the first instance comprises one or more threads of execution, and is capable of performing a write operation on state information stored in non-volatile memory and associated with the software entity;
in response to said detection, invoking an operating system interface to create a quiesced state of the first instance by stopping any in-progress write operations on the state information associated with the first instance of the software entity and by suspending execution of the one or more threads of execution in the first instance;
in response to said quiesced state of the first instance, re-instantiating the software entity to create a second instance of the software entity such that the first and second instances are executing at the same time, wherein the second instance comprises one or more threads of execution, and is capable of performing a write operation on the state information associated with the software entity;
in response to said quiesced state of the first instance, obtaining diagnostic information from the first instance; and
in response to obtaining the diagnostic information, terminating the first instance.
2. The method of claim 1 , further comprising, in response to said quiesced state, invoking the operating system interface to send a signal to a monitoring process signifying that the first instance has been fenced off, wherein said re-instantiation is responsive to the signal.
3. The method of claim 1 , further comprising, in response to said quiesced state, generating a core dump from the first instance.
4. The method of claim 1 , further comprising, in response to said termination, sending a SIGCHLD signal to a monitoring process signifying that the first instance has terminated.
5. A computer program product comprising a computer useable memory device having a computer readable program stored thereon, wherein the computer readable program when executed on a computer causes the computer to:
detect a failure occurring during execution of a first instance of a software entity, wherein the software entity is an application or process and the first instance comprises one or more threads of execution, and is capable of performing a write operation on state information stored in non-volatile memory and associated with the software entity;
in response to said detection, invoke an operating system interface to create a quiesced state of the first instance by stopping any in-progress write operations on the state information associated with the first instance of the software entity and by suspending execution of the one or more threads of execution in the first instance;
in response to said quiesced state of the first instance, re-instantiate the software entity to create a second instance of the software entity such that the first and second instances are executing at the same time, wherein the second instance comprises one or more threads of execution, and is capable of performing a write operation on the state information associated with the software entity;
in response to said quiesced state of the first instance, obtain diagnostic information from the first instance; and
in response to obtaining the diagnostic information, terminate the first instance.
6. The computer program product of claim 5 , wherein the computer readable program when executed on a computer further causes the computer to:
in response to said quiesced state, invoke the operating system interface to send a signal to a monitoring process signifying that the first instance has been fenced off, wherein said re-instantiation is responsive to the signal.
7. The computer program product of claim 5 , wherein the computer readable program when executed on a computer further causes the computer to:
in response to said quiesced state, generate a core dump from the first instance.
8. The computer program product of claim 5 , wherein the computer readable program when executed on a computer further causes the computer to:
in response to said termination, send a SIGCHLD signal to a monitoring process signifying that the first instance has terminated.
9. The computer program product of claim 5 , wherein the computer program product is stored on a computer useable optical storage medium.
10. The computer program product of claim 5 , wherein the computer program product is stored on a hard disk.
11. A system comprising:
non-volatile memory having state information stored therein, wherein the state information is associated with a software entity; and
a processor configured with logic to
detect a failure occurring during execution of a first instance of the software entity, wherein the software entity is an application or process and the first instance comprises one or more threads of execution, and is capable of performing a write operation on the state information,
in response to said detection, invoke an operating system interface to create a quiesced state of the first instance by stopping any in-progress write operations on the first instance of the state information and by suspending execution of the one or more threads of execution in the first instance,
in response to said quiesced state of the first instance, re-instantiate the software entity to create a second instance of the software entity such that the first and second instances are executing at the same time, wherein the second instance comprises one or more threads of execution, and is capable of performing a write operation on the state information,
in response to said quiesced state of the first instance, obtain diagnostic information from the first instance, and
in response to obtaining the diagnostic information, terminate the first instance.
12. The system of claim 11 , wherein the processor is further configured with the logic to:
in response to said quiesced state, invoke the operating system interface to send a signal to a monitoring process signifying that the first instance has been fenced off, wherein said re-instantiation is responsive to the signal.
13. The system of claim 11 , wherein the processor is further configured with the logic to:
in response to said quiesced state, generate a core dump from the first instance.
14. The system of claim 11 , wherein the processor is further configured with the logic to:
in response to said termination, send a SIGCHLD signal to a monitoring process signifying that the first instance has terminated.