Method and apparatus for high availability distributed processing across independent networked computer fault groups
View Patent ↗A plurality of local network groups of computers ( 102 ) are coupled together by a network ( 104 ). Independent processing systems that execute a single operating system are coupled together by a network ( 220 ) to form the local network groups. The independent processing systems may have more than one CPU ( 202 ). One or more of the independent processing systems may share power, cooling and a housing, thereby forming a common fault processor group ( 200 ). An application is written to execute across multiple independent processing systems and common fault processor groups. That is, the application runs in many instances that each run on independent processing systems. The multiple instances of the application provide some measure of high availability by using N+K sparing or the like. The application is for example, call processing or radio control. A processor notification list ( 304 ) keeps track of the independent processing systems that cooperatively provide an application. The independent processing systems monitor state information for independent processing systems on its processor notification lists. Also, the independent processing systems exchange messages periodically with independent processing systems on its processor notification list. The failure to receive a message is noted as a failure that requires recovery. Multiple high availability software systems are permitted to coexist under a single operating system by virtue of a new software layer that coordinates initialization and requires registration of all high availability software.
1. An apparatus for high availability processing comprising:
a plurality of independent processing systems coupled together via a network;
an application that has a plurality of instances, wherein each instance of the plurality of instances executes on one of the plurality of independent processing systems;
a partner notification list stored on each one of the plurality of independent processing systems, the partner notification list indicating which of the plurality of independent processing systems executes an instance of the application;
wherein each one of the plurality of independent processing systems sends messages to each independent processing system on its partner notification list and expects messages from each independent processing system on its partner notification lists;
wherein a failure is detected when an expected message is not received; and
wherein a recovery scheme is implemented in response to detecting the failure and the recovery scheme includes changing the plurality of independent processing systems that execute an instance of the application.
2. The apparatus of claim 1 wherein the recovery scheme includes determining whether a network is accessible.
3. The apparatus of claim 1 further comprising at least one alarm card that is associated with each independent processing system, wherein the at least one alarm card includes an interface to the network.
4. The apparatus of claim 1 wherein the recovery scheme includes sending a message to an alarm card associated with an independent processing system from which a message was expected, but not received.
5. An apparatus for high availability processing comprising:
a plurality of independent processing systems coupled together via a network;
an application that has a plurality of instances, wherein each instance of the plurality of instances executes on one of the plurality of independent processing systems;
a partner notification list stored on each one of the plurality of independent processing systems, the partner notification list indicating which of the plurality of independent processing systems executes an instance of the application;
wherein each one of the plurality of independent processing systems sends messages to each independent processing system on its partner notification list and expects messages from each independent processing system on its partner notification lists;
wherein a failure is detected when an expected message is not received; and
wherein a recovery scheme is executed on each independent processing system that detects the failure in response to detecting the failure and the recovery scheme includes determining whether communication over the network is working.
6. The apparatus of claim 5 wherein the recovery scheme further includes determining whether an alarm card associated with each independent processing system is accessible by each independent processing system.
7. The apparatus of claim 5 wherein if the recovery scheme determines communication over the network is not working, then the recovery scheme determines whether an alarm card associated with each independent processing system is accessible by each independent processing system.
8. The apparatus of claim 7 wherein if the alarm card associated with the each independent processing system is not accessible by the each independent processing system, then the each independent processing systems resets or reboots.
9. The apparatus of claim 8 wherein if the alarm card associated with the each independent processing system is accessible by the each independent processing system, then a communications failure is reported.
10. The apparatus of claim 5 wherein if the recovery scheme determines communication over the network is working, then the recovery scheme request a state of the independent processing system from which the expected message was expected.
11. The apparatus of claim 10 wherein if the state is active, then recovery of the independent processing system from which the expected message was expected is requested.
12. The apparatus of claim 10 wherein if the state is not active, then the not active status is reported.
13. A method for providing high availability to an application, the method comprising the steps of:
providing a plurality of independent processing systems coupled together via a network;
executing an application that has a plurality of instances, wherein each instance of the plurality of instances executes on one of the plurality of independent processing systems;
maintaining a partner notification list on each one of the plurality of independent processing systems, the partner notification list indicating which of the plurality of independent processing systems executes an instance of the application;
each one of the plurality of independent processing systems sending messages to each independent processing system on its partner notification list and expecting messages from each independent processing system on its partner notification lists;
detecting a failure when an expected message is not received; and
executing a recovery scheme on each independent processing system in response to detecting the failure, wherein the recovery scheme includes determining whether communication over the network is working.
14. The method of claim 13 wherein the recovery scheme further includes determining whether an alarm card associated with each independent processing system is accessible by each independent processing system.
15. The method of claim 13 wherein if the recovery scheme determines communication over the network is not working, then the recovery scheme determines whether an alarm card associated with each independent processing system is accessible by each independent processing system.
16. The method of claim 15 wherein if the alarm card associated with the each independent processing system is not accessible by the each independent processing system, then the each independent processing systems resets or reboots.
17. The method of claim 16 wherein if the alarm card associated with the each independent processing system is accessible by the each independent processing system, then a communications failure is reported.
18. The apparatus of claim 13 wherein if the recovery scheme determines communication over the network is working, then the recovery scheme request a state of the independent processing system from which the expected message was expected.
19. The method of claim 18 wherein if the state is active, then recovery of the independent processing system from which the expected message was expected is requested.
20. The method of claim 18 wherein if the state is not active, then the not active status is reported.