Inter-processor failure detection and recovery
An approach to detecting processor failure in a multi-processor environment is disclosed. The approach may include having each CPU in the system responsible for monitoring another CPU in the system. A CPU n reads a timestamp n+1 created by CPU n+1 which CPU n is monitoring from a shared memory location. The CPU n reads its own timestamp n and compares the two timestamps to calculate a delta value. If the delta value is above a threshold, the CPU n determines that CPU n+1 has failed and initiates error handling for the CPUs in the system. One CPU may be designated a master CPU, and be responsible for beginning the error handling process. In such embodiments, the CPU n may initiate error handling by notifying the master CPU that CPU n+1 has failed. If CPU n+1 is the master CPU, the CPU n may take additional steps to initiate error handling, and may broadcast a non-critical interrupt to all CPUs, triggering error handling.
1. An apparatus comprising:
non-transitory computer readable storage medium storing computer readable prongram code executable by a plurality of centaral processing units (CPU), wherein the plurality of CPUs are configured in a ring and each CPU n determines whether a CPU n+1 that is logically adjacent to the CPU n in the ring has failed, the computer readable program code comprising:
a retrieval module of the CPU n configured to retrieve a timestamp n+1 from a shared memory that is shared by the plurality of CPUs, wherein the timestamp n+1 is written to the shared memory by the CPU n+1 , wherein the CPU n is a first core in a multi-core processor and the CPU n+1 is a second core in a multi-core processor, the multi-core processor comprising a plurality of cores;
a comparison module of the CPU n configured to compare the timestamp n+1 to a timestamp n generated by a CPU n checking the CPU n+1 for failure and determine a delta value;
the comparison module of the CPU n further configured to compare the delta value with a threshold value and determine whether the CPU n+1 has failed; and
a detection module of the CPU n configured to, in response to the comparison module determining that the CPU n+1 has failed, initiate error handling for the plurality of CPUs.
2. The apparatus of claim 1 , the computer readable program code comprising further comprising a timestamp module to read the timestamp n from hardware and write the timestamp n to the shared memory.
3. The apparatus of claim 1 , wherein the threshold value is set lower than a system threshold value for the system in which the multi-processor device operates.
4. The apparatus of claim 1 , wherein the CPU n+1 is not a master CPU and the CPU n is not the master CPU, initiating error handling comprising the CPU n notifying the master CPU of the failure on CPU n+1 , and wherein the master CPU causes the plurality of CPUs to perform error handling.
5. The apparatus of claim 1 , wherein the CPU n+1 is a master CPU, the detection module further configured to:
send a non-critical interrupt to CPU n+1 ;
send a critical interrupt to CPU n+1 in response to the CPU n+1 failing to respond to the non-critical interrupt; and
broadcast a group non-critical interrupt to all CPUs in response the CPU n+1 failing to respond to the critical interrupt, wherein the group non-critical interrupt causes the CPUs to perform error handling.
6. The apparatus of claim 1 , wherein each of the plurality of CPUs has a dedicated cache line in the shared memory for writing timestamps.
7. The apparatus of claim 1 , the comparison module further configured to add additional time to the timestamp n prior to comparing the timestamp n+1 to the timestamp n .
8. The apparatus of claim 7 , wherein the additional time accounts for time to move the timestamp n+1 from CPU n+1 to CPU n .
9. The apparatus of claim 1 , wherein the shared memory stores one or more timestamps generated by the plurality of CPUs in a global array.
10. A system comprising:
a shared memory that is shared by a plurality of central processing units (CPUs), wherein the plurality of CPUs are configured in a ring and each CPU n determines whether a CPU n+1 that is logically adjacent to the CPU n in the ring has failed;
the CPU n+1 of the plurality of CPUs configured to write a timestamp n+1 to a global array in the shared memory, wherein the CPU n is a first core in a multi-core processor and the CPU n+1 is a second core in the multi-core processing, the multi-core processor comprising a plurality of cores;
the CPU n of the plurality of CPUs configured to detect a failure in CPU n+1 , detecting a failure comprising the steps of:
retrieving the timestamp n+1 from the shared memory;
comparing the timestamp n+1 to a timestamp n generated by the CPU n and determining a delta value;
comparing the delta value with a threshold value and determining whether the CPU n+1 has failed; and
in response to determining that the CPU n+1 has failed, initiating error handling for the plurality of CPUs.
11. The system of claim 10 , wherein the shared memory and the plurality of CPUs are components of a Fibre Channel Storage Host Adapter.
12. The system of claim 11 , wherein the threshold value is set lower than a system threshold value for the Fibre Channel Storage Host Adapter.
13. The system of claim 10 , wherein the CPU n is configured to read the timestamp n from hardware and write the timestamp n to the global array.
14. A method for detecting processor failure, the method comprising:
retrieving a timestamp n+1 from a shared memory that is shared by a plurality of central processing units (CPUs), wherein the plurality of CPUs are configured in a ring and each CPU n determines whether a CPU n+1 that is logically adjacent to the CPU n in the ring has failed, the timestamp n+1 is written to the shared memory by the CPU n+1 , wherein the CPU n is a first core in a multi-core processor and the CPU n+1 is a second core in a multi-core processor, the multi-core processor comprising a plurality of cores;
comparing by the CPU n , the timestamp n+1 to a timestamp n generated by the CPU n checking the CPU n+1 for failure; and
in response to the difference between timestamp n+1 and timestamp n being larger than a threshold value, the CPU n determining that there is a failure on CPU n+1 and initiating error handling for the plurality of CPUs.
15. The method of claim 14 , further comprising reading the timestamp n from hardware and writing the timestamp n to the shared memory.
16. The method of claim 14 , wherein the CPU n+1 is not a master CPU and the CPU n is not the master CPU, and wherein initiating error handling comprises the CPU n notifying the master CPU of the failure on CPU n+1 , and wherein the master CPU causes the plurality of CPUs to perform error handling.
17. The method of claim 14 , wherein the CPU n+1 is a master CPU, the method further comprising:
sending a non-critical interrupt to CPU n+1 ;
sending a critical interrupt to CPU n+1 in response to the CPU n+1 failing to respond to the non-critical interrupt; and
broadcasting a group non-critical interrupt to all CPUs in response the CPU n+1 failing to respond to the critical interrupt, wherein the group non-critical interrupt causes the CPUs to perform error handling.
18. The method of claim 14 , wherein each of the plurality of CPUs has a dedicated cache line in the shared memory for writing timestamps.
19. The method of claim 14 , further comprising adding additional time to the timestamp n prior to comparing the timestamp n+1 to the timestamp n .