Inter-process communication fault detection and recovery system
An inter-process communication (IPC) system, includes a first client engine, a first server engine, and a broker engine that is coupled to the first client engine. The broker engine initiates a first timer that is configured to reset when traffic is received from the first server engine while the first server engine is registered with the broker engine and coupled to the broker engine via a communication channel. The traffic that causes the first timer to reset includes at least one of: traffic generated by the first client engine to complete a request, and a first server-to-broker heartbeat message generated by the first server engine. The broker engine determines that the first timer has reached a predefined time amount, and in response, removes the registration of the first server engine and removes the communication channel between the broker engine and the first server engine.
1. An inter-process communication (IPC) system comprising an Information Handling System (IHS) comprising a communication system, a processing system coupled to the communication system, and a memory system coupled to the processing system, the IHS being configured to implement:
a first client engine;
a first server engine; and
a broker engine that is coupled to the first client engine, wherein the broker engine is configured to:
initiate a first timer that is configured to reset when traffic is received from the first server engine while the first server engine is registered with the broker engine and coupled to the broker engine via a communication channel, wherein the traffic that causes the first timer to reset includes at least one of: traffic generated by the first client engine to complete a request, and a first server-to-broker heartbeat message generated by the first server engine; and
determine that the first timer has reached a predefined time amount, and in response, remove the registration of the first server engine and remove the communication channel between the broker engine and the first server engine;
provide, to the first server engine, a broker-to-server heartbeat message while the first server engine is registered with the broker engine and the communication channel is established;
wherein the first server engine is configured to:
initiate a second timer that is configured to reset when traffic is received from the broker engine while the broker engine is registered with the first server engine and coupled to the first server engine via the communication channel, wherein the traffic that causes the second timer to reset includes at least one of: a request generated by the first client engine, and the broker-to-server heartbeat message provided by the broker engine; and
determine that the second timer has reached a predefined time amount and, in response, provide a registration request to the broker engine.
2. The IPC system of claim 1 , wherein at least one of the broker-to-server heartbeat message and the first server-to-broker heartbeat message includes a sequence number, and wherein the first server engine is configured to utilize the sequence number to determine whether the broker engine has failed, and wherein the broker engine is configured to utilize the sequence number to determine whether the first server engine has failed.
3. The IPC system of claim 1 , wherein the broker engine is further configured to:
receive a second server-to-broker heartbeat message subsequent to removing the registration of the first server engine and removing the communication channel and, in response, provide a not-registered message to the first server engine; and
receive a registration request from the first server engine and, in response, re-register the first server engine and reestablish the communication channel between the first server engine and the broker engine.
4. The IPC system of claim 1 , wherein the broker engine is configured to:
provide, to the first client engine in response to receiving a client request and the first timer reaching the predefined time amount, a notification that the first server engine is unavailable.
5. The IPC system of claim 1 , wherein the first server-to-broker heartbeat message is generated by the first server engine when the traffic generated by the first client engine to complete the request is not received by the broker engine for a predefined time period.
6. An information handling system (IHS), comprising:
a communication system;
a processing system coupled to the communication system; and
a memory system coupled to the processing system and including instructions that, when executed by the processing system, cause the processing system to provide a broker engine that is configured to:
initiate a first timer that is configured to reset when traffic is received from a first server engine while the first server engine is registered with the broker engine and coupled to the broker engine via a communication channel provided by the communication system, wherein the traffic that causes the first timer to reset includes at least one of: traffic generated by a first client engine coupled to the broker engine through the communication system to complete a request, and a first server-to-broker heartbeat message generated by the first server engine;
determine that the first timer has reached a predefined time amount, and in response, remove the registration of the first server engine and remove the communication channel between the broker engine and the first server engine;
provide, to the first server engine via the communication system, a broker-to-server heartbeat message while the first server engine is registered with the broker engine and the communication channel is established;
receive, from the first server engine, a registration request, wherein the registration request is received in response to the first server engine initiating a second timer that is configured to reset when traffic is received from the broker engine while the broker engine is registered with the first server engine and coupled to the first server engine via the communication channel provided by the communication system, wherein the traffic that causes the second timer to reset includes at least one of: a request generated by the first client engine, and the broker-to-server heartbeat message provided by the broker engine, and determining that the second timer has reached a predefined time amount and, in response, providing a registration request to the broker engine.
7. The IHS of claim 6 , wherein at least one of the broker-to-server heartbeat message and the first server-to-broker heartbeat message includes a sequence number, and wherein the first server engine is configured to utilize the sequence number to determine whether the broker engine has failed, and wherein the broker engine is configured to utilize the sequence number to determine whether the first server engine has failed.
8. The IHS of claim 6 , wherein the broker engine is further configured to:
receive a second server-to-broker heartbeat message subsequent to removing the registration of the first server engine and removing the communication channel and, in response, provide a not-registered message to the first server engine; and
receive a registration request from the first server engine and, in response, re-register the first server engine and reestablish the communication channel between the first server engine and the broker engine.
9. The IHS of claim 6 , wherein the broker engine is configured to:
provide, to the first client engine in response to receiving a client request and the first timer reaching the predefined time amount, a notification that the first server engine is unavailable.
10. The IHS of claim 6 , wherein the first server-to-broker heartbeat message is generated by the first server engine when the traffic generated by the first client engine to complete the request is not received by the broker engine for a predefined time period.
11. A method of fault detection in an inter-process communication system, comprising:
initiating, by a broker engine, a first timer that is configured to reset when traffic is received from a first server engine coupled to the broker engine while the first server engine is registered with the broker engine and coupled to the broker engine via a communication channel, wherein the traffic that causes the first timer to reset includes at least one of: traffic generated by a first client engine coupled to the broker engine to complete a request, and a first server-to-broker heartbeat message generated by the first server engine;
determining, by the broker engine, that the first timer has reached a predefined time amount, and in response, remove the registration of the first server engine and remove the communication channel between the broker engine and the first server engine;
providing, by the broker engine to the first server engine, a broker-to-server heartbeat message while the first server engine is registered with the broker engine and the communication channel is established;
receiving, by the broker engine, registration request, wherein the registration request is provided to the broker engine by the first server engine in response to the first server engine initiating a second timer that is configured to reset when traffic is received from the broker engine while the broker engine is registered with the first server engine and coupled to the first server engine via the communication channel, wherein the traffic that causes the second timer to reset includes at least one of: a request generated by the first client engine, and the broker-to-server heartbeat message provided by the broker engine, and the first server engine determining that the second timer has reached a predefined time amount.
12. The method of claim 11 , wherein at least one of the broker-to-server heartbeat message and the first server-to-broker heartbeat message includes a sequence number, and wherein the first server engine is configured to utilize the sequence number to determine whether the broker engine has failed, and wherein the broker engine is configured to utilize the sequence number to determine whether the first server engine has failed.
13. The method of claim 11 , further comprising:
receiving, by the broker engine, a second server-to-broker heartbeat message subsequent to removing the registration of the first server engine and removing the communication channel and, in response, provide a not-registered message to the first server engine; and
receiving, by the broker engine, a registration request from the first server engine and, in response, re-register the first server engine and reestablish the communication channel between the first server engine and the broker engine.
14. The method of claim 11 , further comprising:
providing, by the broker engine to the first client engine in response to receiving a client request and the first timer reaching the predefined time amount, a notification that the first server engine is unavailable.