IP Library › Granted Patent US 12,430,029
Granted Patent B2
US 12,430,029 · App. 18/530,649 · Granted Sep 30, 2025

Storage system with dynamic fair queue scheduling of host and replication input-output operations

Inventors: Igor Achkinazi (Northborough, MA); Lev Knopov (Brookline, MA)
Assignee: Dell Products L.P.
G06F3/061G06F3/065G06F3/0679
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,430,029
App. No.
18/530,649
Granted
Sep 30, 2025
Kind
B2
Abstract

An apparatus includes at least one processing device comprising a processor coupled to a memory. The processing device is configured to process host input-output operations received in a first storage system from at least one host device, the host input-output operations being placed in a first storage queue of the first storage system to await access to one or more backend storage devices of the first storage system, to process replication input-output operations in the first storage system for delivery to a second storage system, the replication input-output operations being placed in a second storage queue of the first storage system to await access to the one or more backend storage devices of the first storage system, and to dynamically adjust priorities of the respective first and second storage queues for access to the one or more backend storage devices in accordance with one or more priority adjustment criteria.

Claims (65)

1. A method comprising:

processing host input-output operations received in a first storage system from at least one host device, the host input-output operations being placed in a first storage queue of the first storage system to await access to one or more backend storage devices of the first storage system;

processing replication input-output operations in the first storage system for delivery to a second storage system, the replication input-output operations being placed in a second storage queue of the first storage system to await access to the one or more backend storage devices of the first storage system; and

dynamically adjusting priorities of the respective first and second storage queues for access to the one or more backend storage devices in accordance with one or more priority adjustment criteria;

wherein dynamically adjusting priorities of the respective first and second storage queues comprises utilizing wait time measurements generated for at least one of a completion queue associated with the host input-output operations and a replication queue associated with the replication input-output operations to determine an adjustment to be made to the priorities of the respective first and second storage queues;

wherein dynamically adjusting priorities of the respective first and second storage queues further comprises:

generating wait time measurements for respective ones of the first and second storage queues;

generating a wait time measurement for the completion queue;

generating a wait time measurement for the replication queue;

adjusting the wait time measurement for the first storage queue based at least in part on the wait time measurement for the completion queue;

adjusting the wait time measurement for the second storage queue based at least in part on the wait time measurement for the replication queue;

comparing the adjusted wait time measurement for the first storage queue to the adjusted wait time measurement for the second storage queue; and

dynamically adjusting priorities of the respective first and second storage queues based at least in part on a result of the comparing; and

wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

2. The method of claim 1 wherein the first storage system further comprises, in addition to the first and second storage queues, first and second out-bound queues, the first out-bound queue being for outgoing communications directed from the first storage system to the host device and relating to the host input-output operations, the second out-bound queue being for outgoing communications directed from the first storage system to the second storage system and relating to the replication input-output operations.

3. The method of claim 2 wherein dynamically adjusting priorities of the respective first and second storage queues for access to the one or more backend storage devices further comprises adjusting one or more of the priorities based at least in part on wait times of the first and second storage queues and wait times of the first and second out-bound queues.

4. The method of claim 1 wherein dynamically adjusting priorities of the respective first and second storage queues based at least in part on a result of the comparing comprises one of:

responsive to the adjusted wait time measurement for the first storage queue being greater than the adjusted wait time measurement for the second storage queue by more than a first threshold amount, increasing the priority of the first storage queue and decreasing the priority of the second storage queue; and

responsive to the adjusted wait time measurement for the second storage queue being greater than the adjusted wait time measurement for the first storage queue by more than a second threshold amount, increasing the priority of the second storage queue and decreasing the priority of the first storage queue.

5. The method of claim 4 wherein increasing the priority of one of the first and second storage queues comprises increasing the priority by a designated delta value.

6. The method of claim 4 wherein the priority of the first storage queue cannot be decreased below a designated minimum value.

7. The method of claim 1 wherein adjusting the wait time measurement for the first storage queue based at least in part on the wait time measurement for the completion queue comprises summing the respective wait time measurements for the first storage queue and the completion queue.

8. The method of claim 1 wherein adjusting the wait time measurement for the second storage queue based at least in part on the wait time measurement for the replication queue comprises summing the respective wait time measurements for the second storage queue and the replication queue.

9. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code, when executed by at least one processing device comprising a processor coupled to a memory, causes said at least one processing device:

to process host input-output operations received in a first storage system from at least one host device, the host input-output operations being placed in a first storage queue of the first storage system to await access to one or more backend storage devices of the first storage system;

to process replication input-output operations in the first storage system for delivery to a second storage system, the replication input-output operations being placed in a second storage queue of the first storage system to await access to the one or more backend storage devices of the first storage system; and

to dynamically adjust priorities of the respective first and second storage queues for access to the one or more backend storage devices in accordance with one or more priority adjustment criteria;

wherein dynamically adjusting priorities of the respective first and second storage queues comprises utilizing wait time measurements generated for at least one of a completion queue associated with the host input-output operations and a replication queue associated with the replication input-output operations to determine an adjustment to be made to the priorities of the respective first and second storage queues; and

wherein dynamically adjusting priorities of the respective first and second storage queues further comprises:

generating wait time measurements for respective ones of the first and second storage queues;

generating a wait time measurement for the completion queue;

generating a wait time measurement for the replication queue;

adjusting the wait time measurement for the first storage queue based at least in part on the wait time measurement for the completion queue;

adjusting the wait time measurement for the second storage queue based at least in part on the wait time measurement for the replication queue;

comparing the adjusted wait time measurement for the first storage queue to the adjusted wait time measurement for the second storage queue; and

dynamically adjusting priorities of the respective first and second storage queues based at least in part on a result of the comparing.

10. An apparatus comprising:

at least one processing device comprising a processor coupled to a memory;

said at least one processing device being configured:

to process host input-output operations received in a first storage system from at least one host device, the host input-output operations being placed in a first storage queue of the first storage system to await access to one or more backend storage devices of the first storage system;

to process replication input-output operations in the first storage system for delivery to a second storage system, the replication input-output operations being placed in a second storage queue of the first storage system to await access to the one or more backend storage devices of the first storage system; and

to dynamically adjust priorities of the respective first and second storage queues for access to the one or more backend storage devices in accordance with one or more priority adjustment criteria;

wherein dynamically adjusting priorities of the respective first and second storage queues comprises utilizing wait time measurements generated for at least one of a completion queue associated with the host input-output operations and a replication queue associated with the replication input-output operations to determine an adjustment to be made to the priorities of the respective first and second storage queues; and

wherein dynamically adjusting priorities of the respective first and second storage queues further comprises:

generating wait time measurements for respective ones of the first and second storage queues;

generating a wait time measurement for the completion queue;

generating a wait time measurement for the replication queue;

adjusting the wait time measurement for the first storage queue based at least in part on the wait time measurement for the completion queue;

adjusting the wait time measurement for the second storage queue based at least in part on the wait time measurement for the replication queue;

comparing the adjusted wait time measurement for the first storage queue to the adjusted wait time measurement for the second storage queue; and

dynamically adjusting priorities of the respective first and second storage queues based at least in part on a result of the comparing.

11. The apparatus of claim 10 wherein said at least one processing device comprises at least a portion of the first storage system.

12. The apparatus of claim 10 wherein dynamically adjusting priorities of the respective first and second storage queues comprises:

assigning values to respective first and second time slices of a designated processing interval to the first and second storage queues; and

varying the assigned values of the first and second time slices over time.

13. The apparatus of claim 12 wherein the values assigned to the first and second time slices comprise respective percentages of a time duration of the processing interval.

14. The apparatus of claim 12 wherein the values assigned to the first and second time slices comprise respective numbers of input-output operations associated with the processing interval.

15. The apparatus of claim 10 wherein at least one of the wait time measurements comprises at least one of a simple moving average of a first designated number of previous input-output operations and an exponential moving average of a second designated number of previous input-output operations.

16. The apparatus of claim 10 wherein dynamically adjusting priorities of the respective first and second storage queues based at least in part on a result of the comparing comprises one of:

responsive to the adjusted wait time measurement for the first storage queue being greater than the adjusted wait time measurement for the second storage queue by more than a first threshold amount, increasing the priority of the first storage queue and decreasing the priority of the second storage queue; and

responsive to the adjusted wait time measurement for the second storage queue being greater than the adjusted wait time measurement for the first storage queue by more than a second threshold amount, increasing the priority of the second storage queue and decreasing the priority of the first storage queue.

17. The apparatus of claim 16 wherein increasing the priority of one of the first and second storage queues comprises increasing the priority by a designated delta value.

18. The apparatus of claim 16 wherein the priority of the first storage queue cannot be decreased below a designated minimum value.

19. The apparatus of claim 10 wherein adjusting the wait time measurement for the first storage queue based at least in part on the wait time measurement for the completion queue comprises summing the respective wait time measurements for the first storage queue and the completion queue.

20. The apparatus of claim 10 wherein adjusting the wait time measurement for the second storage queue based at least in part on the wait time measurement for the replication queue comprises summing the respective wait time measurements for the second storage queue and the replication queue.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 6, 2023
From: ACHKINAZI, IGOR; KNOPOV, LEV
To: DELL PRODUCTS L.P.
Reel/Frame 065780/0083 →
Continuity (1)
Related Publication 20250190110A1 · Jun 12, 2025
References Cited (38)
US 10310760B1 · Dreier et al. · 2019 [cited by applicant]
US 10893105B1 · Bono et al. · 2021 [cited by applicant]
US 11550511B2 · Mallick et al. · 2023 [cited by applicant]
US 20030056038A1 · Cochran · 2003 [cited by examiner]
US 20030149773A1 · Harbin et al. · 2003 [cited by applicant]
US 20090154472A1 · Chung et al. · 2009 [cited by applicant]
US 20130198312A1 · Tamir et al. · 2013 [cited by applicant]
US 20130226887A1 · Braam et al. · 2013 [cited by applicant]
US 20150012607A1 · Cayton et al. · 2015 [cited by applicant]
US 20150143053A1 · Quimbey · 2015 [cited by examiner]
US 20170177222A1 · Singh et al. · 2017 [cited by applicant]
US 20200019521A1 · Solanki et al. · 2020 [cited by applicant]
US 20200026606A1 · Farnum et al. · 2020 [cited by applicant]
US 20210266651A1 · Luo · 2021 [cited by examiner]
US 20220283962A1 · Hwang · 2022 [cited by examiner]
US 20220350755A1 · Hahn · 2022 [cited by examiner]
US 20220374167A1 · Mallick et al. · 2022 [cited by applicant]
US 20230195656A1 · Walker · 2023 [cited by examiner]
US 20230229314A1 · Chen et al. · 2023 [cited by applicant]
US 20230297238A1 · Mallick et al. · 2023 [cited by applicant]
US 20230325074A1 · Achkinazi et al. · 2023 [cited by applicant]
US 20230325089A1 · Rasal · 2023 [cited by examiner]
US 20230325114A1 · Achkinazi et al. · 2023 [cited by applicant]
Storpool Storage, “Demystifying: What is NVMeOF?” https://storpool.com/blog/demystifying-what-is-nvmeof, Sep. 12, 2017, 4 pages. [cited by applicant]
VMware, “VMware ESX Server,” Product Datasheet, 2007, 4 pages. [cited by applicant]
Wikipedia, “Host Adapter,” https://en.wikipedia.org/wiki/Host_adapter, Jul. 19, 2021, 4 pages. [cited by applicant]
Wikipedia, “iSCSI,” https://en.wikipedia.org/wiki/ISCSI, Dec. 22, 2021, 10 pages. [cited by applicant]
Wikipedia, “NVM Express,” https://en.wikipedia.org/wiki/NVM_Express, Jan. 13, 2022, 18 pages. [cited by applicant]
A. S. Gillis, “NVMe Over Fabrics (NVMe-oF),” https://searchstorage.techtarget.com/definition/NVMe-over-Fabrics-Nonvolatile-Memory-Express-over-Fabrics?vgnextfmt=print, Jan. 15, 2020, 5 pages. [cited by applicant]
Wikipedia, “Remote Direct Memory Access,” https://en.wikipedia.org/wiki/Remote_direct_memory_access, Jan. 30, 2021, 3 pages. [cited by applicant]
M. Hoyt, “ScaleIO Tech Overview and Concepts: SDS-SAN vs SDS-Array,” https://www.thinkahead.com/TheLAB/scaleio-tech-overview-concepts-sds-san-vs-sds-array/, Apr. 5, 2017, 16 pages. [cited by applicant]
EMC Corporation, “EMC ScaleIO Architectural and Functional Overview,” EMC White Paper, Dec. 2013, 13 pages. [cited by applicant]
Dell EMC, “Dell EMC VxFlex OS: Networking Best Practices and Design Considerations,” Dell EMC White Paper, Jul. 2018, 38 pages. [cited by applicant]
NVM Express, “NVM Express Base Specification, Revision 2.0c,” NVM Express, Oct. 4, 2022, 458 pages. [cited by applicant]
Mellanox Technologies, “RoCE vs. iWARP Competitive Analysis,” White Paper, Feb. 2017, 6 pages. [cited by applicant]
EMC Corporation, “EMC ScaleIO Design Considerations and Best Practices,” EMC White Paper, Jun. 2016, 30 pages. [cited by applicant]
U.S. Appl. No. 17/964,560 filed in the name of Igor Achkinazi et al. filed Oct. 12, 2022, and entitled “Host-Based Locality Determination Using Locality Log Pages.” [cited by applicant]
U.S. Appl. No. 18/335,240 filed in the name of Igor Achkinazi et al. filed Jun. 15, 2023, and entitled “Storage System with Automated Filtering of Discovery Information Utilizing Specified Configuration Domains.” [cited by applicant]