IP Library › Granted Patent US 10,725,684
Granted Patent B2
US 10,725,684 · App. 14/502,327 · Granted Jul 28, 2020

Method and apparatus for cost-based load balancing for port selection

Inventors: Ajith Balakrishnan (Bangalore, IN); Felix Shvaiger (Brighton, MA); Alexandr Veprinsky (Brookline, MA); Arieh Don (Newton, MA)
Assignee: EMC IP Holding Company LLC
G06F3/0635G06F3/0613G06F3/0659G06F3/0683
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,725,684
App. No.
14/502,327
Filed
Sep 30, 2014
Granted
Jul 28, 2020
Kind
B2
Art Unit
2181
USPC
710/38
Abstract

Example embodiments relate to a method, a system, and a computer program product for load balancing for port selection. The method includes determining a processing load for each storage port in a plurality of storage ports having variable processing power and calculating a delay characteristic for each storage port of the plurality of storage ports according to its processing load. A command then may be sent to a selected storage port of the plurality of storage ports according to the delay characteristics and a policy.

Claims (67)

1. A method comprising:

determining a processing load for each storage port in a plurality of storage ports wherein each storage port has an amount of dynamically assigned processing cores;

calculating a delay characteristic for each storage port of the plurality of storage ports according to its processing load; and

sending a command to a selected storage port of the plurality of storage ports according to the delay characteristics and a policy;

wherein determining the processing load for each storage port comprises:

sending a query to one of a set of two or more of the plurality of storage ports associated with a given front-end adapter of a storage system; and

receiving, from one of the set of two or more storage ports associated with the given front-end adapter of the storage system, processing load information for each storage port in the set of two or more storage ports associated with the given front-end adapter of the storage system; and

wherein determining the processing load for each storage port further comprises at least one of:

querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for command volume metadata associated with each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system; and

querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for command processing time metadata associated with each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system.

2. The method of claim 1 wherein determining the processing load for each storage port comprises:

querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for the command volume metadata associated with each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system; and

querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for the command processing time metadata associated with each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system.

3. The method of claim 1 wherein querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for the command volume metadata further comprises:

querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for a current queue depth of each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system; and

querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for a previous number of commands received in a previous period for each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system.

4. The method of claim 3 wherein querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for the command processing time metadata further comprises querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for an average time to process a single command on each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system.

5. The method of claim 4 wherein the average time to process a single command on each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system comprises a moving average of processing times for the previous period for each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system.

6. The method of claim 1 wherein calculating the delay characteristic for each storage port of the plurality of storage ports according to its processing load comprises, for a given storage port:

determining a first portion of the delay characteristic attributable to network propagation delay; and

calculating a second portion of the delay characteristic attributable to processing commands in a queue at the given storage port and a number of commands sent to the given storage port.

7. The method of claim 6 wherein calculating the second portion of the delay characteristic attributable to processing commands in the queue at the given storage port and the number of commands sent to the given storage port comprises calculating the second portion of the delay characteristic according to an average time to process a single command on the given storage port.

8. The method of claim 7 wherein calculating the second portion of the delay characteristic according to the average time to process a single command on the given storage port comprises calculating a first subportion of the second portion of the delay characteristic according to a sent number of commands sent to the given storage port in a previous period from one or more host ports other than a selected host port.

9. The method of claim 8 wherein calculating the first subportion of the second portion of the delay characteristic according to the sent number of commands sent to the given storage port in the previous period from the one or more host ports other than the selected host port comprises:

determining a received previous number of commands received at the given storage port from all host ports in the previous period; and

determining a sent previous number of commands sent from the selected host port to the given storage port in the previous period.

10. The method of claim 8 further comprising calculating a second subportion of the second portion of the delay characteristic according to a current number of commands sent from the selected host port to the given storage port in a current period.

11. A system comprising:

a plurality of hosts each comprising respective host ports; and

a storage system comprising a plurality of storage ports having variable processing power; and

computer program code that when executed on a processor of a computer causes the computer to perform the operations of:

determining a processing load for each storage port in a plurality of storage ports wherein each storage port has an amount of dynamically assigned processing cores;

calculating a delay characteristic for each storage port of the plurality of storage ports according to its processing load; and

sending a command to a selected storage port of the plurality of storage ports according to the delay characteristics and a policy;

wherein determining the processing load for each storage port comprises:

sending a query to one of a set of two or more of the plurality of storage ports associated with a given front-end adapter of a storage system; and

receiving, from one of the set of two or more storage ports associated with the given front-end adapter of the storage system, processing load information for each storage port in the set of two or more storage ports associated with the given front-end adapter of the storage system; and

wherein determining the processing load for each storage port further comprises at least one of:

querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for command volume metadata associated with each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system; and

querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for command processing time metadata associated with each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system.

12. The system of claim 11 wherein determining the processing load for each storage port comprises:

querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for the command volume metadata associated with each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system; and

querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for the command processing time metadata associated with each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system.

13. The system of claim 11 wherein querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for command volume metadata further comprises:

querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for a current queue depth of each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system; and

querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for a previous number of commands received in a previous period for each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system.

14. The system of claim 13 wherein querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for the command processing time metadata further comprises querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for an average time to process a single command on each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system.

15. The system of claim 14 wherein the average time to process a single command on each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system comprises a moving average of processing times for the previous period for each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system.

16. The system of claim 11 wherein calculating the delay characteristic for each storage port of the plurality of storage ports according to its processing load comprises, for a given storage port:

determining a first portion of the delay characteristic attributable to network propagation delay; and

calculating a second portion of the delay characteristic attributable to processing commands in a queue at the given storage port and a number of commands sent to the given storage port.

17. The system of claim 16 wherein calculating the second portion of the delay characteristic attributable to processing commands in the queue at the given storage port and the number of commands sent to the given storage port comprises calculating the second portion of the delay characteristic according to an average time to process a single command on the given storage port.

18. The system of claim 17 wherein calculating the second portion of the delay characteristic according to the average time to process a single command on the given storage port comprises calculating a first subportion of the second portion of the delay characteristic according to a sent number of commands sent to the given storage port in a previous period from one or more of the host ports other than a selected one of the host ports.

19. The system of claim 18 wherein calculating the first subportion of the second portion of the delay characteristic according to the sent number of commands sent to the given storage port in the previous period from the one or more host ports other than the selected host port comprises:

determining a received previous number of commands received at the given storage port from all host ports in the previous period; and

determining a sent previous number of commands sent from the selected host port to the given storage port in the previous period.

20. The system of claim 18 wherein calculating the second portion of the delay characteristic according to the average time to process a single command on the given storage port further comprises calculatin 1 g a second subportion of the second portion of the delay characteristic according to a current number of commands sent from the selected host port to the given storage port in a current period.

21. A computer program product including a non-transitory computer-readable storage medium having computer program code encoded thereon that when executed on a processor of a computer causes the computer to perform multipathing according to storage port processing load, the computer program code comprising:

computer program code for determining a processing load for each storage port in a plurality of storage ports wherein each storage port has an amount of dynamically assigned processing cores;

computer program code for calculating a delay characteristic for each storage port of the plurality of storage ports according to its processing load; and

computer program code for sending a command to a selected storage port of the plurality of storage ports according to the delay characteristics and a policy;

wherein determining the processing load for each storage port comprises:

sending a query to one of a set of two or more of the plurality of storage ports associated with a given front-end adapter of a storage system; and

receiving, from one of the set of two or more storage ports associated with the given front-end adapter of the storage system, processing load information for each storage port in the set of two or more storage ports associated with the given front-end adapter of the storage system; and

wherein determining the processing load for each storage port further comprises at least one of:

querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for command volume metadata associated with each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system; and

querying said one of the set of two or more storage ports associated with the given front-end adapter of the storage system for command processing time metadata associated with each of the storage ports in the set of two or more storage ports associated with the given front-end adapter of the storage system.

Assignments (10)
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (053546/0001) Recorded Jun 23, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC IP HOLDING COMPANY LLC
Reel/Frame 071642/0001 →
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001) Recorded May 20, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.); DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC); EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.); SCALEIO LLC
Reel/Frame 061753/0001 →
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001) Recorded Apr 26, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.); DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC); EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.); SCALEIO LLC
Reel/Frame 061324/0001 →
RELEASE OF SECURITY INTEREST Recorded Nov 3, 2021
From: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH
To: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL, L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; WYSE TECHNOLOGY L.L.C.
Reel/Frame 058216/0001 →
SECURITY AGREEMENT Recorded Apr 22, 2020
From: CREDANT TECHNOLOGIES INC.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; FORCE10 NETWORKS, INC.; WYSE TECHNOLOGY L.L.C.; EMC IP HOLDING COMPANY LLC
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A.
Reel/Frame 053546/0001 →
SECURITY AGREEMENT Recorded Mar 21, 2019
From: CREDANT TECHNOLOGIES, INC.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; FORCE10 NETWORKS, INC.; WYSE TECHNOLOGY L.L.C.; EMC IP HOLDING COMPANY LLC
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A.
Reel/Frame 049452/0223 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 29, 2016
From: EMC CORPORATION
To: EMC IP HOLDING COMPANY LLC
Reel/Frame 040203/0001 →
SECURITY AGREEMENT Recorded Sep 21, 2016
From: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; SPANNING CLOUD APPS LLC; WYSE TECHNOLOGY L.L.C.
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
Reel/Frame 040136/0001 →
SECURITY AGREEMENT Recorded Sep 21, 2016
From: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; SPANNING CLOUD APPS LLC; WYSE TECHNOLOGY L.L.C.
To: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT
Reel/Frame 040134/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 30, 2014
From: BALAKRISHNAN, AJITH; SHVAIGER, FELIX; VEPRINSKY, ALEXANDR; DON, ARIEH
To: EMC CORPORATION
Reel/Frame 033854/0069 →
Continuity (1)
Related Publication 20160092136A1 · Mar 31, 2016