Intelligent dual inline memory module thermal controls for maximum uptime
A dual in-line memory module (DIMM) thermal control system for intelligent DIMM thermal controls for maximum uptime may include a memory subsystem. The memory subsystem may include a first DIMM and a first serial presence detect (SPD) module associated with the first DIMM. The DIMM thermal control system may also include a baseboard management controller (BMC). The BMC may, when a first DIMM failure of the first DIMM may be detected, record a first failure event in a first failure events log of the first SPD module. The first failure event may comprise the first DIMM failure and associated first thermal telemetry data of the first DIMM. The BMC may also adjust DIMM thermal control settings to reduce temperature of the first DIMM based on the first failure events log including at least the first failure event.
1. A dual in-line memory module (DIMM) thermal control system, comprising:
a memory subsystem comprising:
a first DIMM;
a second DIMM;
a first serial presence detect (SPD) module associated with the first DIMM;
a basic input/output system (BIOS) configured to:
prior to a first DIMM failure of the first DIMM being detected:
scan the first SPD module for a first failure events log; and
a baseboard management controller (BMC) configured to:
when the first DIMM failure of the first DIMM is detected:
record a first failure event in the first failure events log of the first SPD module, wherein the first failure event comprises the first DIMM failure and associated first thermal telemetry data of the first DIMM;
adjust first DIMM thermal control settings to reduce a first temperature of the first DIMM based on the first failure events log including at least the first failure event; and
adjust second DIMM thermal control settings to set a second temperature of the second DIMM based on the first failure events log including at least the first failure event.
2. The DIMM thermal control system of claim 1 , wherein the BIOS is further configured to:
send an indicator to the BMC indicating that a second DIMM failure of a second failure event has been detected, wherein the second failure event further comprises second thermal telemetry data of the first DIMM associated with the second DIMM failure, and wherein
the BMC is further configured to:
set the first DIMM thermal control settings to set the first temperature of the first DIMM based on the indicator received from the BIOS indicating that the second DIMM failure of the second failure event has been detected and the first failure events log including at least the second failure event.
3. The DIMM thermal control system of claim 1 , further comprising:
a central processing unit (CPU) coupled to the memory subsystem, wherein
the BMC is further configured to:
set CPU thermal control settings to set the temperature of the CPU based on the first failure events log including at least the first failure event.
4. The DIMM thermal control system of claim 1 , wherein the first failure events log of the first SPD module further comprises:
a DIMM temperature during a warning threshold;
a DIMM temperature during a critical threshold;
a number of DIMM excursions exceeding an excursions limit;
a number of times a DIMM temperature has exceed a temperature threshold;
a number of warning excursions within one or more warning temperature ranges; and
a number of critical excursions within one or more critical temperature ranges.
5. The DIMM thermal control system of claim 1 , wherein the adjustment of the first DIMM thermal control settings to reduce the first temperature of the first DIMM further comprises one or more of a reduction of a DIMM fan control target by a fan control reduction amount, a reduction of a DIMM closed loop thermal throttling (CLTT) high temperature value by a CLTT reduction amount, and a reduction of a CPU cooling target by a cooling reduction amount.
6. The DIMM thermal control system of claim 1 , wherein the adjustment of the first DIMM thermal control settings to reduce first temperature of the first DIMM is further based on health parameters of the first DIMM.
7. The DIMM thermal control system of claim 6 , wherein the health parameters comprise:
a number of spare DIMMs remaining;
past temperature excursions of the first DIMM; and
on-die error correcting code (ECC) error occurrences of the first DIMM.
8. The DIMM thermal control system of claim 1 , wherein the first DIMM failure comprises one or more of:
a correctable error; and
an uncorrectable error.
9. The DIMM thermal control system of claim 1 , wherein after the first failure event has been recorded in the first failure events log, the BMC is further configured to:
when a consecutive error count of the first failure events log exceeds a consecutive error count threshold, reset the consecutive error count to zero.
10. A method, comprising:
prior to a first DIMM failure of a first DIMM of a memory subsystem of a DIMM thermal control system being detected:
scanning, by a basic input/output system (BIOS) of the memory subsystem, a first serial presence detect (SPD) module of the memory subsystem for a first failure events log, the first SPD module associated with the first DIMM; and
when the first DIMM failure of the first DIMM is detected:
recording, by a baseboard management controller (BMC) of the DIMM thermal control system, a first failure event in the first failure events log of the first SPD module, wherein the first failure event comprises the first DIMM failure and associated first thermal telemetry data of the first DIMM;
adjusting, by the BMC, first DIMM thermal control settings to reduce a first temperature of the first DIMM based on the first failure events log including at least the first failure event; and
adjusting, by the BMC, second DIMM thermal control settings to set a second temperature of a second DIMM based on the first failure events log including at least the first failure event.
11. The method of claim 10 , the method further comprising:
sending, by the BIOS, an indicator to the BMC indicating that a second DIMM failure of a second failure event has been detected, wherein the second failure event further comprises second thermal telemetry data of the first DIMM associated with the second DIMM failure;
setting, by the BMC, the first DIMM thermal control settings to set the first temperature of the first DIMM based on the indicator received from the BIOS indicating that the second DIMM failure of the second failure event has been detected and the first failure events log including at least the second failure event.
12. The method of claim 10 , the method further comprising:
setting, by the BMC, CPU thermal control settings to set the temperature of a central processing unit (CPU) of the DIMM thermal control system based on the first failure events log including at least the first failure event.
13. The method of claim 10 , wherein the first failure events log of the first SPD module further comprises:
a DIMM temperature during a warning threshold;
a DIMM temperature during a critical threshold;
a number of DIMM excursions exceeding an excursions limit;
a number of times a DIMM temperature has exceed a temperature threshold;
a number of warning excursions within one or more warning temperature ranges; and
a number of critical excursions within one or more critical temperature ranges.
14. The method of claim 10 , wherein adjusting the first DIMM thermal control settings to reduce the first temperature of the first DIMM further comprises one or more of reducing a DIMM fan control target by a fan control reduction amount, reducing a DIMM closed loop thermal throttling (CLTT) high temperature value by a CLTT reduction amount, and reducing a CPU cooling target by a cooling reduction amount.
15. The method of claim 10 , wherein adjusting the first DIMM thermal control settings to reduce first temperature of the first DIMM is further based on health parameters of the first DIMM.
16. The method of claim 15 , wherein the health parameters comprise:
a number of spare DIMMs remaining;
past temperature excursions of the first DIMM; and
on-die error correcting code (ECC) error occurrences of the first DIMM.
17. The method of claim 10 , wherein the first DIMM failure comprises one or more of:
a correctable error; and
an uncorrectable error.
18. The method of claim 10 , wherein after the first failure event has been recorded in the first failure events log, when a consecutive error count of the first failure events log exceeds a consecutive error count threshold, resetting the consecutive error count to zero.