Failure isolation in a communication system
View Patent ↗A computer system including a communication fabric initiates a forced diagnostic to isolate and identify genuine error conditions which are discerned from sympathetic error conditions. Error counters are only incremented for each genuine error condition, precluding the need to set error counter threshold artificially high. Recovery events are logged in a recovery table and recovery actions are only initiated after the diagnoses processes is complete. This prevents duplication of recovery actions and the unnecessary implementation of low-level recovery actions when they will be followed by higher-level recovery actions.
1. A method of isolating failures in a communication fabric, the communication fabric including a processing device, a threshold counter, a fabric controller, a host adapter, and a host, the method comprising the steps of:
detecting an initiating event in the processing device;
initiating a warmstart procedure causing the fabric controller to cease communication with the host adapter;
pausing a predetermined period to allow a plurality of error messages including a first error message indicative of a failure of a first component and a second error message indicative of a reaction of a second component to a condition caused the failure of the first component to arrive at a processing device, said processing device including a software subcomponent; and
evaluating the plurality of error messages to discern a source of an error,
wherein the second error message is a component timeout, and wherein evaluating the plurality of error messages to discern a source of an error comprises determining that the host adapter is not a source of an error.
2. A method of isolating failures in a communication fabric, the communication fabric including a processing device, a threshold counter, a fabric controller, a host adapter, and a host, the method comprising the steps of:
detecting an initiating event in the processing device;
initiating a warmstart procedure causing the fabric controller to cease communication with the host adapter;
pausing a predetermined period to allow a plurality of error messages including a first error message indicative of a failure of a first component and a second error message indicative of a reaction of a second component to a condition caused the failure of the first component to arrive at a processing device, said processing device including a software subcomponent; and
evaluating the plurality of error messages to discern a source of an error,
wherein the second error message is an adapter warmstart timeout, and wherein evaluating the plurality of error messages to discern a source of an error comprises determining that the host adapter is a source of an error.
3. A method of isolating failures in a communication fabric, the communication fabric including a processing device, a threshold counter, a fabric controller, a host adapter, and a host, the method comprising the steps of:
detecting an initiating event in the processing device;
initiating a warmstart procedure causing the fabric controller to cease communication with the host adapter;
pausing a predetermined period to allow a plurality of error messages including a first error message indicative of a failure of a first component and a second error message indicative of a reaction of a second component to a condition caused the failure of the first component to arrive at a processing device, said processing device including a software subcomponent; and
evaluating the plurality of error messages to discern a source of an error,
wherein the second error message is a fabric interrupt, and wherein evaluating the plurality of error messages to discern a source of an error comprises determining that the host adapter has performed a fabric protocol violation.
4. A method of isolating failures in a communication fabric, the communication fabric including a processing device, a threshold counter, a fabric controller, a host adapter, and a host, the method comprising the steps of:
detecting an initiating event in the processing device;
initiating a warmstart procedure causing the fabric controller to cease communication with the host adapter;
pausing a predetermined period to allow a plurality of error messages including a first error message indicative of a failure of a first component and a second error message indicative of a reaction of a second component to a condition caused the failure of the first component to arrive at a processing device, said processing device including a software subcomponent; and
evaluating the plurality of error messages to discern a source of an error,
wherein the second error message is an adapter failure, and wherein evaluating the plurality of error messages to discern a source of an error comprises transmitting a heartbeat message to the host adapter, receiving a component timeout, and determining that the host adapter is a source of an error.
5. A method of isolating failures in a communication fabric, the communication fabric including a processing device, a threshold counter, a fabric controller, a host adapter, and a host, the method comprising the steps of:
detecting an initiating event in the processing device;
initiating a warmstart procedure causing the fabric controller to cease communication with the host adapter;
pausing a predetermined period to allow a plurality of error messages including a first error message indicative of a failure of a first component and a second error message indicative of a reaction of a second component to a condition caused the failure of the first component to arrive at a processing device, said processing device including a software subcomponent; and
evaluating the plurality of error messages to discern a source of an error,
wherein the second error message is an adapter interrupt, and wherein evaluating the plurality of error messages to discern a source of an error comprises determining that the host adapter is a source of an error.