IP Library › Granted Patent US 10,331,144
Granted Patent B2
US 10,331,144 · App. 15/017,604 · Granted Jun 25, 2019

Liquid flow control management for shared infrastructure servers

Inventors: Austin Michael Shelnutt (Leander, TX); Paul W. Vancil (Austin, TX); John R. Stuewe (Round Rock, TX); Charles Edward Smith (Austin, TX)
Assignee: Dell Products, L.P.
G05D7/0629G05B15/02H05K7/20736H05K7/20772H05K7/20781H05K7/20836
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,331,144
App. No.
15/017,604
Filed
Feb 6, 2016
Granted
Jun 25, 2019
Kind
B2
Art Unit
2116
USPC
700/282
Abstract

A computer-implemented method controls liquid cooling of a direct injection liquid-cooled (DL) rack information handling system (RIHS). The method includes receiving, at a liquid cooling control subsystem, an incoming cooling liquid supply flow rate corresponding to an incoming cooling liquid supply being supplied to the DL RIHS. A maximum flow rate cap is calculated for each of the LC nodes. The maximum flow rate cap is transmitted to a controller for each of the LC nodes. The controller triggers each of the LC nodes to adjust the associated flow rate for that LC node to correspond to the received maximum flow rate cap for that node.

Claims (99)

1. A computer-implemented method for controlling liquid-based cooling of liquid-cooled (LC) nodes in a LC rack information handling system (RIHS), the method comprising:

receiving, at a liquid cooling control subsystem, an incoming cooling liquid supply flow rate corresponding to an amount of incoming cooling liquid being supplied to the LC RIHS;

calculating a maximum flow rate cap for each block of LC nodes;

transmitting the maximum flow rate cap to a controller for each block of LC nodes;

triggering the controller for each block of LC nodes to adjust an associated flow rate for that block of LC nodes to correspond to the received maximum flow rate cap for that block;

determining if the maximum flow rate cap calculated for any one of the LC nodes is outside of a pre-determined range; and

in response to determining that at least one of the maximum flow rate caps is outside of the pre-determined range, setting the maximum flow rate cap for each of the LC nodes to a default flow rate.

2. The method of claim 1 , wherein each block of LC nodes includes at least one liquid intake control valve that controls a flow rate of incoming cooling liquid to the block of LC nodes and wherein calculating the maximum flow rate cap for each block of LC nodes comprises:

triggering each controller to set a respective liquid control valve to a first percentage of a fully open valve position corresponding to a respective first flow rate;

measuring a second flow rate of liquid flowing through the liquid intake control valve and transmitting the measured second flow rate to the liquid cooling control subsystem; and

wherein calculating the maximum flow rate cap is based at least partially on the measured second flow rate and at least partially on the incoming cooling liquid supply flow rate.

3. The method of claim 2 , wherein calculating the maximum flow rate cap for each of the blocks of LC nodes further comprises:

calculating a first sum of second flow rates for all blocks of LC nodes;

calculating an average flow rate from the first sum and the incoming cooling liquid supply flow rate; and

calculating the maximum flow rate cap by dividing the average flow rate by a number of blocks of LC nodes within the LC RIHS.

4. The method of claim 1 , wherein the liquid cooling control subsystem includes a rack-level liquid infrastructure controller (RLIC) and the controller for each block is a block liquid controller (BLC), the method further comprising:

determining, via each BLC, a third flow rate corresponding to actual cooling requirements of an associated block of LC nodes and transmitting the third flow rate to the RLIC;

determining if the third flow rate for any of the LC nodes is less than a corresponding maximum flow rate cap;

in response to determining that at least one third flow rate is less than a corresponding maximum flow rate cap, calculating a decreased flow rate cap for each block with a third flow rate that is less than a corresponding maximum flow rate cap;

transmitting the decreased maximum flow rate cap to a corresponding BLC to trigger the corresponding BLC to adjust a setting of the block valve to allow the flow rate to be reduced to at or below the decreased maximum flow rate cap.

5. The method of claim 4 , further comprising:

determining, via the BLC, whether a third flow rate desired for cooling a corresponding block is greater than a current maximum flow rate cap assigned to the block;

in response to the third flow rate being greater than the current maximum flow rate cap, calculating an increased flow rate cap for the block based on one or more variables associated with operational settings of the block of LC nodes;

in response to the increased flow rate cap being available due to a decrease in a previous maximum flow rate cap of another block, transmitting the increased flow rate cap to the BLC to trigger the BLC to increase a flow rate through the block valve up to the increased flow rate cap, wherein a sum of all maximum flow caps allocated to through all blocks of the RIHS does not exceed a maximum flow of the incoming cooling liquid supply flow rate, and wherein the incoming cooling liquid supply flow is dynamically re-allocated from blocks requiring less than an allocated maximum flow rate cap to blocks requiring more than a corresponding maximum flow rate cap.

6. The method of claim 1 , wherein the LC nodes are grouped into at least one block, each of the at least one blocks having an associated block liquid controller that is communicatively coupled to a rack level liquid infrastructure controller, the method further comprising:

receiving, at each of the block liquid controllers, temperature data associated with each of the LC nodes within each of the blocks;

identifying a highest magnitude temperature from among the temperature data;

retrieving a first operating temperature set point associated with the LC node having the highest magnitude temperature;

calculating a fourth flow rate for each of the at least one block based on the highest magnitude temperature value and the first operating temperature set point; and

triggering each of the BLCs to move their respective liquid control valves to a first position corresponding to the fourth flow rate such that sufficient cooling liquid is provided to maintain the first operating set point for the LC nodes within each of the at least one block of the LC RIHS.

7. The method of claim 6 , wherein triggering the liquid control valves to move to the first position further comprises:

generating pulse width modulated (PWM) signals that are proportional to the fourth flow rate for each of the blocks; and

transmitting the PWM signals to respective liquid control valves.

8. A liquid-cooled (LC) rack information handling system (RIHS) comprising:

a rack having a plurality of LC nodes;

a liquid cooling subsystem associated with the rack, the liquid cooling subsystem including:

a first flow rate meter for measuring an incoming cooling liquid supply being supplied to the LC RIHS;

a rack level liquid infrastructure controller, the rack level liquid infrastructure controller communicatively coupled to the LC nodes and to the first flow rate meter, wherein the rack level liquid infrastructure controller:

receives an incoming cooling liquid supply flow rate corresponding to an amount of an incoming cooling liquid being supplied to the LC RIHS;

calculates a maximum flow rate cap for each of the LC nodes, wherein to calculate the maximum flow rate cap for each of the LC nodes, the rack level liquid infrastructure controller:

calculates a first sum of second flow rates for all the LC nodes;

calculates an average flow rate of the first sum and the incoming cooling liquid supply flow rate; and

calculates the maximum flow rate cap by dividing the average flow rate by a number of LC nodes;

transmits the maximum flow rate cap to a controller for each of the LC nodes; and

triggers the controller for each of the LC nodes to adjust an associated flow rate for that LC node to correspond to the received maximum flow rate cap for that node.

9. The liquid cooled rack information handling system of claim 8 , further comprising:

at least one liquid control valve associated with each of the LC nodes that controls incoming cooling liquid to respective ones of the LC nodes; and

wherein the rack level liquid infrastructure controller:

triggers each of the LC nodes to set a respective liquid control valve to a first percentage of a fully open valve position corresponding to a respective first flow rate;

triggers each of the LC nodes to measure a second flow rate of liquid flowing through the LC node and to transmit the measured second flow rate to the liquid cooling control subsystem; and

wherein calculating the maximum flow rate cap is based at least partially on the measured second flow rate and at least partially on the incoming cooling liquid supply flow rate.

10. The liquid cooled rack information handling system of claim 8 , wherein the rack level liquid infrastructure controller:

determines if the maximum flow rate cap calculated for any one of the LC nodes is outside of a pre-determined range; and

in response to determining that at least one of the maximum flow rate caps is outside of the pre-determined range, sets the maximum flow rate cap for each of the LC nodes to a default flow rate.

11. The liquid cooled rack information handling system of claim 8 , wherein the rack level liquid infrastructure controller:

triggers each of the LC nodes to measure a third flow rate through each of their respective LC nodes and to transmit the measured third flow rates to the rack level liquid infrastructure controller;

determines if the third flow rate for each of the LC nodes is less than their corresponding maximum flow rate cap;

in response to determining that at least one third flow rate is less than their corresponding maximum flow rate cap, calculates a decreased flow rate cap for the LC nodes that have a third flow rate less than their corresponding maximum flow rate cap and calculating an increased flow rate cap for each of the LC nodes that have third flow rates greater than their corresponding maximum flow rate cap, wherein the total maximum flow rate caps for the LC nodes are less than the incoming cooling liquid supply flow rate for the LC RIHS;

transmits respective decreased flow rate caps and respective increased flow rate caps to each of the corresponding LC nodes; and

triggers the corresponding LC nodes to adjust each of their respective flow rates to correspond to the received flow rate caps, wherein the incoming cooling liquid supply flow is re-allocated from LC nodes that require less than their corresponding maximum flow rate cap to LC nodes that require more than their corresponding maximum flow rate cap.

12. The liquid cooled rack information handling system of claim 8 , wherein the LC nodes are grouped into at least one block, each of the at least one blocks having an associated block liquid controller that is communicatively coupled to the rack level liquid infrastructure controller and wherein each of the block liquid controllers:

receives temperature data associated with each of the LC nodes within each of the blocks;

identifies a highest magnitude temperature from among the temperature data;

retrieves a first operating temperature set point associated with the LC node having the highest magnitude temperature;

calculates a fourth flow rate for each of the at least one block based on the highest magnitude temperature value and the first operating temperature set point; and

triggers each of the LC nodes to move their respective liquid control valves to a first position corresponding to the fourth flow rate such that sufficient cooling liquid is provided to each of the LC nodes within each of the at least one block of the LC RIHS.

13. The liquid cooled rack information handling system of claim 12 , wherein to trigger the liquid control valves to move to the first position the block liquid controller:

generates pulse width modulated (PWM) signals that are proportional to the fourth flow rate for each of the blocks; and

transmits the PWM signals to respective liquid control valves.

14. A liquid cooling system for a liquid cooled (LC) rack information handling system (RIHS) comprising:

a liquid cooling subsystem including a first flow rate meter for measuring an incoming cooling liquid supply being supplied to the LC RIHS and a rack level liquid infrastructure controller, the rack level liquid infrastructure controller communicatively coupled to the first flow rate meter, wherein the rack level liquid infrastructure controller:

receives an incoming cooling liquid supply flow rate corresponding to an amount of an incoming cooling liquid being supplied to the LC RIHS;

calculates a maximum flow rate cap for each of a plurality of LC nodes, wherein to calculate the maximum flow rate cap for each of the LC nodes, the rack level liquid infrastructure controller:

calculates a first sum of second flow rates for all the LC nodes;

calculates an average flow rate of the first sum and the incoming cooling liquid supply flow rate; and

calculates the maximum flow rate cap by dividing the average flow rate by a number of LC nodes;

transmits the maximum flow rate cap to a controller for each of the LC nodes; and

triggers the controller for each of the LC nodes to adjust an associated flow rate for that LC node to correspond to the received maximum flow rate cap for that node.

15. The liquid cooling system of claim 14 , further comprising:

at least one liquid control valve associated with each of the LC nodes that controls incoming cooling liquid to respective ones of the LC nodes; and

wherein the firmware further configures the rack level liquid infrastructure controller to:

trigger each of the LC nodes to set a respective liquid control valve to a first percentage of a fully open valve position corresponding to a respective first flow rate;

trigger each of the LC nodes to measure a second flow rate of liquid flowing through the LC node and to transmit the measured second flow rate to the liquid cooling control subsystem; and

wherein calculating the maximum flow rate cap is based at least partially on the measured second flow rate and at least partially on the incoming cooling liquid supply flow rate.

16. The liquid cooled rack information handling system of claim 14 , wherein the firmware further configures the rack level liquid infrastructure controller to:

determine if the maximum flow rate cap calculated for any one of the LC nodes is outside of a pre-determined range; and

in response to determining that at least one of the maximum flow rate caps is outside of the pre-determined range, set the maximum flow rate cap for each of the LC nodes to a default flow rate.

17. The liquid cooled rack information handling system of claim 14 , wherein the firmware further configures the rack level liquid infrastructure controller to:

trigger each of the LC nodes to measure a third flow rate through each of their respective LC nodes and to transmit the measured third flow rates to the rack level liquid infrastructure controller;

determine if the third flow rate for each of the LC nodes is less than their corresponding maximum flow rate cap;

in response to determining that at least one of the third flow rates is less than their corresponding maximum flow rate cap, calculate a decreased flow rate cap for the LC nodes that have a third flow rate less than their corresponding maximum flow rate cap and calculating an increased flow rate cap for each of the LC nodes that have third flow rates greater than their corresponding maximum flow rate cap, wherein the total maximum flow rate caps for the LC nodes are less than the incoming cooling liquid supply flow rate for the LC RIHS;

transmit respective decreased flow rate caps and respective increased low rate caps to each of the corresponding LC nodes; and

trigger the corresponding LC nodes to adjust each of their respective flow rates to correspond to the received flow rate caps, wherein the incoming cooling liquid supply flow is re-allocated from LC nodes that require less than their corresponding maximum flow rate cap to LC nodes that require more than their corresponding maximum flow rate cap.

18. The liquid cooled rack information handling system of claim 14 , wherein the LC nodes are grouped into at least one block, each of the at least one blocks having an associated block liquid controller that is communicatively coupled to the rack level liquid infrastructure controller and wherein each of the block liquid controllers have firmware executing thereon that configures the block liquid controller to:

receive temperature data associated with each of the LC nodes within each of the blocks;

identify a highest magnitude temperature from among the temperature data;

retrieve a first operating temperature set point associated with the LC node having the highest magnitude temperature;

calculate a fourth flow rate for each of the at least one block based on the highest magnitude temperature value and the first operating temperature set point; and

trigger each of the LC nodes to move their respective liquid control valves to a first position corresponding to the fourth flow rate such that sufficient cooling liquid is provided to each of the LC nodes within each of the at least one block of the LC RIHS.

Assignments (15)
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (053546/0001) Recorded Jun 23, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC IP HOLDING COMPANY LLC
Reel/Frame 071642/0001 →
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001) Recorded May 20, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.); DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC); EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.); SCALEIO LLC
Reel/Frame 061753/0001 →
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001) Recorded Apr 26, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.); DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC); EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.); SCALEIO LLC
Reel/Frame 061324/0001 →
RELEASE OF SECURITY INTEREST Recorded Nov 3, 2021
From: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH
To: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL, L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; WYSE TECHNOLOGY L.L.C.
Reel/Frame 058216/0001 →
SECURITY AGREEMENT Recorded Apr 22, 2020
From: CREDANT TECHNOLOGIES INC.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; FORCE10 NETWORKS, INC.; WYSE TECHNOLOGY L.L.C.; EMC IP HOLDING COMPANY LLC
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A.
Reel/Frame 053546/0001 →
SECURITY AGREEMENT Recorded Mar 21, 2019
From: CREDANT TECHNOLOGIES, INC.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; FORCE10 NETWORKS, INC.; WYSE TECHNOLOGY L.L.C.; EMC IP HOLDING COMPANY LLC
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A.
Reel/Frame 049452/0223 →
SECURITY AGREEMENT Recorded Sep 21, 2016
From: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; SPANNING CLOUD APPS LLC; WYSE TECHNOLOGY L.L.C.
To: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT
Reel/Frame 040134/0001 →
SECURITY AGREEMENT Recorded Sep 21, 2016
From: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; SPANNING CLOUD APPS LLC; WYSE TECHNOLOGY L.L.C.
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
Reel/Frame 040136/0001 →
RELEASE OF REEL 038664 FRAME 0908 (NOTE) Recorded Sep 14, 2016
From: BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS COLLATERAL AGENT
To: DELL SOFTWARE INC.; DELL PRODUCTS L.P.; SECUREWORKS, CORP.; WYSE TECHNOLOGY L.L.C.
Reel/Frame 040027/0390 →
RELEASE OF REEL 038665 FRAME 0041 (TL) Recorded Sep 14, 2016
From: BANK OF AMERICA, N.A., AS COLLATERAL AGENT
To: DELL SOFTWARE INC.; DELL PRODUCTS L.P.; SECUREWORKS, CORP.; WYSE TECHNOLOGY L.L.C.
Reel/Frame 040028/0375 →
RELEASE OF REEL 038665 FRAME 0001 (ABL) Recorded Sep 13, 2016
From: BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT
To: DELL SOFTWARE INC.; DELL PRODUCTS L.P.; SECUREWORKS, CORP.; WYSE TECHNOLOGY L.L.C.
Reel/Frame 040021/0348 →
SUPPLEMENT TO PATENT SECURITY AGREEMENT (NOTES) Recorded May 11, 2016
From: DELL SOFTWARE INC.; WYSE TECHNOLOGY, L.L.C.; DELL PRODUCTS L.P.
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS FIRST LIEN COLLATERAL AGENT
Reel/Frame 038664/0908 →
SUPPLEMENT TO PATENT SECURITY AGREEMENT (TERM LOAN) Recorded May 11, 2016
From: DELL PRODUCTS L.P.; DELL SOFTWARE INC.; WYSE TECHNOLOGY, L.L.C.
To: BANK OF AMERICA, N.A., AS COLLATERAL AGENT
Reel/Frame 038665/0041 →
SUPPLEMENT TO PATENT SECURITY AGREEMENT (ABL) Recorded May 11, 2016
From: DELL PRODUCTS L.P.; DELL SOFTWARE INC.; WYSE TECHNOLOGY, L.L.C.
To: BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 038665/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 27, 2016
From: SHELNUTT, AUSTIN MICHAEL; VANCIL, PAUL W.; STUEWE, JOHN R.; SMITH, CHARLES EDWARD
To: DELL PRODUCTS, L.P.
Reel/Frame 038108/0267 →
Continuity (4)
Provisional Application 62270584 · Dec 22, 2015
Provisional Application 62270574 · Dec 21, 2015
Provisional Application 62270563 · Dec 21, 2015
Related Publication 20170177007A1 · Jun 22, 2017
Cited By (7)
US 12,200,901 US 12,309,965 US 12,464,675 US 12,516,752 US 12,520,452 US 12,557,241 US 12,563,696