Proactively addressing fan or fan tray failures
View Patent ↗Fan or fan tray failures are currently handled within the scope of the information handling system that has suffered the failure. In such cases, the addressing such issues may be difficult or impossible to do without completely shutting down the device. In one or more embodiments, by announcing the failure to one or more protocols, which allows the handling of such a failure event at a topological level rather that purely at the device level, the impact to the device as well as to the overall traffic in the topology may be drastically mitigated.
1. A method for handling a fan or fan tray failure in an information handling system (IHS), the method comprising:
monitoring status of one or more fans or fan trays of the information handling system; and
responsive to receiving a notification of a failure of a fan or fan tray failure of the information handling system:
notifying one or more protocols of the information handling system of the failure to trigger alerting, via the one or more protocols, one or more information handling system communicatively coupled to the information handling system that has the failure to take one or more actions to adjust traffic to the information handling system with the failure; and
sending one or more notifications to the one or more information handling system communicatively coupled to the information handling system with the failure to take one or more actions to adjust traffic to the information handling system with the failure.
2. The method of claim 1 wherein at least one of the one or more actions comprises having traffic blocked to the information handling system with the failure.
3. The method of claim 1 wherein at least one of the one or more actions comprises adjusting a hashing or a weighting to have less traffic directed to the information handling system with the failure.
4. The method of claim 1 further comprising:
monitoring one or more temperatures of the information handling system with the failure;
assigning a severity level based upon at least one of the one or more temperatures; and
including an indicator of the severity level in at least one of the one or more notifications to at least one of the one or more information handling system communicatively coupled to the information handling system with the failure, in which the indicator of the severity level affects which action is taken by at least one of the one or more information handling system communicatively coupled to the information handling system with the failure.
5. The method of claim 4 wherein the severity further comprising:
a low severity level, which indicates to a notification-receiving information handling system from the one or more information handling system communicatively coupled to the information handling system with the failure that the information handling system with the failure may maintain its existing traffic flows but that the notification-receiving information handling system should not send any new traffic flows to the information handling system with the failure;
a medium severity level, which indicates to a notification-receiving information handling system from the one or more information handling system communicatively coupled to the information handling system with the failure that the notification-receiving information handling system should divert at least some existing traffic flows that are currently going to the information handling system with the failure to one or more alternate paths that do not utilize the information handling system with the failure; and
a high severity level, which indicates to a notification-receiving information handling system from the one or more information handling system communicatively coupled to the information handling system with the failure that the notification-receiving information handling system should reroute all traffic to paths that do not utilize the information handling system with the failure.
6. The method of claim 1 further comprising:
monitoring one or more temperatures of the information handling system with the failure, monitoring one or more fans or fan trays of the information handling system with the failure, or both; and
responsive to a change in condition at the information handling system, sending one or more update notifications to at least one of the one or more information handling system communicatively coupled to the information handling system.
7. The method of claim 6 wherein the change of condition is that the failure has been remedied and the method further comprises:
sending one or more notifications to the one or more information handling system communicatively coupled to the information handling system to take one or more actions to resume normal traffic processing.
8. An information handling system comprising:
one or more processors;
a plurality of fans elements comprising one or more fans, one or more fan trays, or both; and
a chassis manager configured to cause steps to be performed comprising:
monitoring status of one or more fans or fan trays of the information handling system; and
responsive to receiving a notification of a failure of a fan or fan tray failure of the information handling system:
notifying one or more protocols of the information handling system of the failure to trigger alerting, via the one or more protocols, one or more information handling system communicatively coupled to the information handling system that has the failure to take one or more actions to adjust traffic to the information handling system with the failure; and
sending one or more notifications to the one or more information handling system communicatively coupled to the information handling system with the failure to take one or more actions to adjust traffic to the information handling system with the failure.
9. The system of claim 8 wherein at least one of the one or more actions comprises having traffic blocked to the information handling system with the failure.
10. The system of claim 8 wherein at least one of the one or more actions comprises adjusting a hashing or a weighting to have less traffic directed to the information handling system with the failure.
11. The system of claim 8 further comprising:
one or more temperature sensors; and
wherein the chassis manager is further configured to cause steps to be performed comprising:
monitoring one or more temperatures of the information handling system with the failure;
assigning a severity level based upon at least one of the one or more temperatures; and
including an indicator of the severity level in at least one of the one or more notifications to at least one of the one or more information handling system communicatively coupled to the information handling system with the failure, in which the indicator of the severity level affects which action is taken by at least one of the one or more information handling system communicatively coupled to the information handling system with the failure.
12. The system of claim 11 wherein the severity further comprising:
a low severity level, which indicates to a notification-receiving information handling system from the one or more information handling system communicatively coupled to the information handling system with the failure that the information handling system with the failure may maintain its existing traffic flows but that the notification-receiving information handling system should not send any new traffic flows to the information handling system with the failure;
a medium severity level, which indicates to a notification-receiving information handling system from the one or more information handling system communicatively coupled to the information handling system with the failure that the notification-receiving information handling system should divert at least some existing traffic flows that are currently going to the information handling system with the failure to one or more alternate paths that do not utilize the information handling system with the failure; and
a high severity level, which indicates to a notification-receiving information handling system from the one or more information handling system communicatively coupled to the information handling system with the failure that the notification-receiving information handling system should reroute all traffic to paths that do not utilize the information handling system with the failure.
13. The system of claim 8 wherein the chassis manager is further configured to cause steps to be performed comprising:
monitoring one or more temperatures of the information handling system with the failure, monitoring one or more fans or fan trays of the information handling system with the failure, or both; and
responsive to a change in condition at the information handling system, sending one or more update notifications to at least one of the one or more information handling system communicatively coupled to the information handling system.
14. The system of claim 13 wherein the change of condition is that the failure has been remedied and the chassis manager is further configured to cause steps to be performed comprising:
sending one or more notifications to the one or more information handling system communicatively coupled to the information handling system to take one or more actions to resume normal traffic processing.
15. An information handling system comprising:
one or more processors;
a plurality of fans elements comprising one or more fans, one or more fan trays, or both;
one or more temperature sensors; and
a chassis manager configured to cause steps to be performed comprising:
monitoring status of one or more fans or fan trays of the information handling system;
monitoring temperature using at least one of the one or more temperature sensors; and
responsive to receiving a notification of a failure of a fan or fan tray failure of the information handling system:
notifying one or more protocols of the information handling system of the failure to trigger alerting, via the one or more protocols, one or more information handling system communicatively coupled to the information handling system that has the failure to take one or more actions to adjust traffic to the information handling system with the failure; and
sending one or more notifications to the one or more information handling system communicatively coupled to the information handling system with the failure to take one or more actions to adjust traffic to the information handling system with the failure.
16. The system of claim 15 wherein at least one of the one or more actions comprises having traffic blocked to the information handling system with the failure.
17. The system of claim 15 wherein at least one of the one or more actions comprises adjusting a hashing or a weighting to have less traffic directed to the information handling system with the failure.
18. The system of claim 15 further wherein the chassis manager is further configured to cause steps to be performed comprising:
assigning a severity level based upon at least one of the one or more temperatures; and
including an indicator of the severity level in at least one of the one or more notifications to at least one of the one or more information handling system communicatively coupled to the information handling system with the failure, in which the indicator of the severity level affects which action is taken by at least one of the one or more information handling system communicatively coupled to the information handling system with the failure.
19. The system of claim 18 wherein the severity further comprising:
a low severity level, which indicates to a notification-receiving information handling system from the one or more information handling system communicatively coupled to the information handling system with the failure that the information handling system with the failure may maintain its existing traffic flows but that the notification-receiving information handling system should not send any new traffic flows to the information handling system with the failure;
a medium severity level, which indicates to a notification-receiving information handling system from the one or more information handling system communicatively coupled to the information handling system with the failure that the notification-receiving information handling system should divert at least some existing traffic flows that are currently going to the information handling system with the failure to one or more alternate paths that do not utilize the information handling system with the failure; and
a high severity level, which indicates to a notification-receiving information handling system from the one or more information handling system communicatively coupled to the information handling system with the failure that the notification-receiving information handling system should reroute all traffic to paths that do not utilize the information handling system with the failure.
20. The system of claim 15 wherein the chassis manager is further configured to cause steps to be performed comprising:
monitoring one or more temperatures of the information handling system with the failure, monitoring one or more fans or fan trays of the information handling system with the failure, or both; and
responsive to a change in condition at the information handling system, sending one or more update notifications to at least one of the one or more information handling system communicatively coupled to the information handling system.