Window-based prority tagging of IOPs in a distributed storage system
A distributed storage schemes manages implementation of QoS targets for IOPs across compute nodes executing applications, primary storage nodes storing a primary copy of a logical storage volume, and clone storage nodes. On the compute node, a maximum priority is assigned to a minimum number of IOPs in a queue within a time window from a time of receipt of a last unexecuted IOP. Other IOPs are assigned a minimum priority. On the storage node, maximum priority IOPs are assigned to high priority queues, from which IOPs are executed first, and low priority IOPs are assigned to low priority queues. Methods for determining the capacity of storage nodes and allocating storage requests are also disclosed.
1. A system comprising one or more processing devices and one or more memory devices operably coupled to the one or more memory devices, the one or more memory devices storing executable code effective to cause the one or more processing devices to:
receive a plurality of input/output operations (IOPs); and
for each IOP
assign a priority to the each IOP according to a priority associated with a storage volume referenced in the each IOP; and
transmit the each IOP tagged with the priority over a network for execution,
wherein the storage volume of the each IOP belongs to a queue group of a plurality of queue groups, each queue group of the plurality of queue groups having a queue belonging thereto, wherein the executable code is further effective to cause the one or more processing devices to:
for each IOP, add the each IOP to a corresponding queue belonging to the queue group to which the storage volume referenced by the each IOP belongs;
transmit the each IOP and the priority from the queue for execution according to a first-in-first-out approach; and
for each IOP, after execution of the each IOP, remove the each IOP from the corresponding queue
wherein the executable code is further effective to cause the one or more processing devices to, for each queue belonging to one of the queue groups of the plurality of queue groups;
assign a first priority to a first portion of the IOPs in the each queue that were received within a time threshold from receipt of an oldest unexecuted IOP in the each queue; and
assign a second priority to any other IOPs in the each queue that were received after the time threshold from receipt of the oldest unexecuted IOP in the each queue, the second priority being lower than the first priority.
2. The system of claim 1 , wherein the executable code is further effective to cause the one or more processing devices to, for each queue belonging to one of the queue groups of the plurality of queue groups:
for each IOP, add the each IOP to the corresponding queue only when a number of IOPs in the corresponding queue is less than a maximum IOP value.
3. The system of claim 1 , wherein the executable code is further effective to cause the one or more processing devices to, for each queue belonging to one of the queue groups of the plurality of queue groups:
(a) beginning at the oldest unexecuted IOP in the each queue, assign the first priority to IOPs in the queue received within the time threshold from receipt of the oldest unexecuted IOP until a number of IOPs assigned the first priority is equal to a minimum IOP value; and
(b) assign the second priority to any remaining IOPs in the each queue not assigned the first priority.
4. The system of claim 3 , wherein the executable code is further effective to cause the one or more processing devices to repeat (a) and (b) periodically for each queue belonging to one of the queue groups of the plurality of queue groups.
5. The system of claim 4 , wherein the executable code is further effective to cause the one or more processing devices to, transmit an updated priority for each unexecuted IOP in the each queue that is changed according to repetition of (a) and (b).
6. The system of claim 1 , further comprising one or more storage nodes, each storage node of the one or more storage nodes further programmed to, for each queue group of at least a portion of the plurality of queue groups:
for each IOP received by the each storage node that corresponds to the each queue group, add the each IOP to a first queue if the each IOP is assigned the first priority and adding the each IOP to a second queue if the each IOP is assigned the second priority; and
execute IOPs within the second queue only when the first queue is empty.
7. The system of claim 6 , wherein each storage node is further programmed to:
select a selected queue group from the at least the portion of the plurality of queue groups according to queue priorities associated with the at least the portion of the plurality of queue groups; and
execute an IOP from one of the first and second queues of the selected queue group.
8. The system of claim 7 , wherein each storage node is further programmed to, for each queue group of the at least the portion of the plurality of queue groups:
assign a unique range of token values to the each queue group, a number of values in the unique range of token values corresponding to the queue priority of the each queue group;
execute an IOP from one of the first and second queues of the each queue group when a token generator outputs a value within the unique range of token values.
9. A method comprising:
receiving, by a computing device comprising one or more processing devices, a plurality of input/output operations (IOPs);
for each IOP
assigning, by the computing device, a priority to the each IOP according to a priority associated with a storage volume referenced in the each IOP; and
transmitting, by the computing device, the each IOP tagged with the priority for execution,
wherein the storage volume of the each IOP belongs to a queue group of a plurality of queue groups, each queue group of the plurality of queue groups having a queue belonging thereto, wherein the method further comprises:
for each IOP, adding, by the computing device, the each IOP to a corresponding queue belonging to the queue group to which the storage volume referenced by the each IOP belongs;
transmitting, by the computing device, the each IOP and the priority from the queue for execution according to a first-in-first-out approach; and
for each IOP, after execution of the each IOP, removing, by the computing device, the each IOP from the corresponding queue; and
wherein the method further comprises, for each queue belonging to one of the queue groups of the plurality of queue groups:
assigning, by the computing device, a first priority to a first portion of the IOPs in the each queue that were received within a time threshold from receipt of an oldest unexecuted IOP in the each queue; and
assigning, by the computing device, a second priority to any other IOPs in the each queue that were received after the time threshold from receipt of the oldest unexecuted IOP in the each queue, the second priority being lower than the first priority.
10. The method of claim 9 , further comprising, for each queue belonging to one of the queue groups of the plurality of queue groups:
for each IOP, adding, by the computing device, the each IOP to the corresponding queue only when a number of IOPs in the corresponding queue is less than a maximum IOP value.
11. The method of claim 9 , further comprising, for each queue belonging to one of the queue groups of the plurality of queue groups:
(a) beginning at the oldest unexecuted IOP in the each queue, assigning, by the computing device, the first priority to IOPs in the queue received within the time threshold from receipt of the oldest unexecuted IOP until a number of IOPs assigned the first priority is equal to a minimum IOP value; and
(b) assigning, by the computing device, the second priority to any remaining IOPs in the each queue not assigned the first priority.
12. The method of claim 11 , further comprising repeating, by the computing device, (a) and (b) periodically for each queue belonging to one of the queue groups of the plurality of queue groups.
13. The method of claim 12 , further comprising transmitting, by the computing device, an updated priority for each unexecuted IOP in the each queue that is changed according to repetition of (a) and (b).
14. The method of claim 9 , further comprising:
providing one or more storage nodes;
by each storage node of the one or more storage nodes, for each queue group of at least a portion of the plurality of queue groups
for each IOP received by the storage node that corresponds to the each queue group, adding the each IOP to a first queue if the each IOP is assigned the first priority and adding the each IOP to a second queue if the each IOP is assigned the second priority; and
executing IOPs within the second queue only when the first queue is empty.
15. The method of claim 14 , further comprising, by each storage node of the one or more storage nodes:
selecting a selected queue group from the at least the portion of the plurality of queue groups according to queue priorities associated with the at least the portion of the plurality of queue groups; and
executing an IOP from one of the first and second queues of the selected queue group.
16. The method of claim 15 , further comprising, by each storage node of the one or more storage nodes, for each queue group of the at least the portion of the plurality of queue groups:
assigning a unique range of token values to the each queue group, a number of values in the unique range of token values corresponding to the queue priority of the each queue group;
executing an IOP from one of the first and second queues of the each queue group when a token generator outputs a value within the unique range of token values.