I/O scheduling and load balancing across the multiple nodes of a clustered environment
Write operations are scheduled for multiple nodes in a shared storage cluster that supports volume replication. Requests are received from nodes for allocation of space for write operations in a replication log. In response to a received request, the current capacity of the requesting node to manage a backlog can be determined. The amount of space in the replication log allocated to the node is then calibrated to the node's capacity, thereby preventing self-throttling. A separate priority can be assigned to each volume, and space in the replication log assigned to each volume based on its priority. Nodes can target synchronous and other latency sensitive operations to higher priority volumes. A single global queue can be maintained to schedule write operations for all nodes, thereby providing a fair scheduling. A separate local queue can be maintained for each node, thereby providing specific levels of preference to specific nodes.
1. A computer-implemented method comprising:
receiving, at a first node of a plurality of nodes in a cluster, an allocation request for space in a replication log, wherein
the first node is configured to receive the allocation request from a second node of the plurality of nodes, and
the replication log stores information describing one or more outstanding write operations;
in response to receiving the allocation request, determining an amount of storage to allocate in the replication log, wherein
the determining is based, at least in part, on one or more parameters, and
the one or more parameters comprise at least one of
a backlog capacity of the second node, or
a scheduling priority for a data volume; and
allocating the amount of storage in the replication log.
2. The computer-implemented method of claim 1 , wherein
the cluster comprises shared storage, and
the second node is configured to perform a write operation by writing first to the replication log and subsequently to the shared storage.
3. The computer-implemented method of claim 2 , wherein
an order of write operations to the replication log and an order of write operations to the shared storage are the same.
4. The computer-implemented method of claim 3 , wherein
the cluster is configured to preserve write-order fidelity by virtue of the cluster being configured to utilize the order of write operations to the replication log to replicate the data volume of the shared storage to a secondary site.
5. The computer-implemented method of claim 1 , wherein the determining the amount of storage to allocate in the replication log comprises:
determining the backlog capacity of the second node, wherein
the backlog capacity of the second node represents a number of outstanding write operations supported by the second node without throttling additional write operations, and
the additional write operations are write operations in addition to the outstanding write operations.
6. The computer-implemented method of claim 5 , wherein the determining the amount of storage to allocate in the replication log further comprises:
limiting the amount of storage allocated in the replication log to an upper limit that reflects a number of simultaneously outstanding write operations supported by the second node.
7. The computer-implemented method of claim 1 , further comprising:
determining a current capacity of the second node from the allocation request, wherein
the current capacity of the second node is the backlog capacity of the second node determined from the allocation request.
8. The computer-implemented method of claim 7 , further comprising:
periodically polling each of the plurality of nodes for information indicative of the current capacity of the second node.
9. The computer-implemented method of claim 1 , wherein the cluster comprises shared storage, the shared storage comprises the data volume, and the method further comprises:
assigning the scheduling priority to the data volume.
10. The computer-implemented method of claim 9 , wherein the shared storage comprises a plurality of data volumes, the plurality of data volumes comprises the data volume, and the method further comprises:
assigning a scheduling priority to each data volume of the plurality of data volumes.
11. The computer-implemented method of claim 10 , wherein the allocating comprises:
allocating storage in the replication log for the each data volume based on the scheduling priority of the each data volume.
12. The computer-implemented method of claim 11 , wherein
a write operation to a first data volume of the plurality of data volumes is processed at a higher level of priority than a write operation to a second data volume of the plurality of data volumes, if a scheduling priority of the first data volume is higher than a scheduling priority of the second data volume.
13. The computer-implemented method of claim 10 , further comprising:
receiving information indicating an adjustment to make to a scheduling priority assigned to a specific data volume of the plurality of data volumes; and
adjusting the scheduling priority assigned to the specific data volume as indicated by information.
14. The computer-implemented method of claim 10 , further comprising:
receiving one or more allocation requests, wherein
the one or more allocation requests are received from one or more nodes of the plurality of nodes,
each of the one or more allocation requests is a request, from a corresponding one of the one or more nodes, for allocation of storage for one or more write operations in the replication log, and
the each of the one or more allocation requests comprises information indicative of a priority of the one or more write operations corresponding to the each of the one or more allocation requests; and
scheduling each of the one or more write operations corresponding to the each of the one or more allocation requests at a level of priority indicated in the one or more allocation requests corresponding to the each of the one or more write operations.
15. A computer program product comprising:
a plurality of instructions, comprising
a first set of instructions, executable on a computer system, configured to
receive, at a first node of a plurality of nodes in a cluster, an allocation request for space in a replication log, wherein
the first node is configured to receive the allocation request from a second node of the plurality of nodes, and
the replication log is configured to store information describing one or more outstanding write operations,
a second set of instructions, executable on the computer system, configured to, in response to receiving the allocation request, determine an amount of storage to allocate in the replication log, wherein
the amount of storage is determined based, at least in part, on one or more parameters, and
the one or more parameters comprise at least one of
a backlog capacity of the second node, or
a scheduling priority for a data volume, and
a third set of instructions, executable on the computer system, configured to allocate the amount of storage in the replication log; and
a computer-readable storage medium, wherein the instructions are encoded in the computer-readable storage medium.
16. The computer program product of claim 15 , wherein
the cluster comprises shared storage,
the second node is configured to perform a write operation by writing first to the replication log and subsequently to the shared storage, and
an order of write operations to the replication log and an order of write operations to the shared storage are the same.
17. The computer program product of claim 15 , wherein the second set of instructions comprises:
a first subset of instructions, executable on the computer system, configured to determine the backlog capacity of the second node, wherein
the backlog capacity of the second node represents a number of outstanding write operations supported by the second node without throttling additional write operations, and
the additional write operations are write operations in addition to the outstanding write operations.
18. The computer program product of claim 15 , wherein the cluster comprises shared storage, the shared storage comprises a plurality of data volumes, the plurality of data volumes comprises the data volume, and the instructions further comprise:
a fourth set of instructions, executable on the computer system, configured to assign a scheduling priority to each data volume of the plurality of data volumes,
wherein
the third set of instructions comprise
a first subset of instructions, executable on the computer system, configured to allocate storage in the replication log for the each data volume based on the scheduling priority of the each data volume.
19. A computer system comprising:
one or more processors;
a computer-readable storage medium coupled to the one or more processors; and
a plurality of instructions, encoded in the computer-readable storage medium and configured to cause the one or more processors to
receive, at a first node of a plurality of nodes in a cluster, an allocation request for space in a replication log, wherein
the first node is configured to receive the allocation request from a second node of the plurality of nodes, and
the replication log is configured to store information describing one or more outstanding write operations;
in response to receiving the allocation request, determine an amount of storage to allocate in the replication log, wherein
the amount of storage is determined based, at least in part, on one or more parameters, and
the one or more parameters comprise at least one of
a backlog capacity of the second node, or
a scheduling priority for a data volume; and
allocate the amount of storage in the replication log.
20. The computer system of claim 19 , wherein the plurality of instructions configured to cause at least one processor of the one or more processors to determine the amount of storage to allocate in the replication log is further configured to cause the at least one processor to:
determine the backlog capacity of the second node, wherein
the backlog capacity of the second node represents a number of outstanding write operations supported by the second node without throttling additional write operations, and
the additional write operations are write operations in addition to the outstanding write operations.
21. The computer system of claim 19 , wherein the cluster comprises shared storage, the shared storage comprises a plurality of data volumes, the plurality of data volumes comprises the data volume, and the plurality of instructions is further configured to cause the at least one processor to:
assign a scheduling priority to each data volume of the plurality of data volumes, wherein
the plurality of instructions configured to cause the at least one processor to allocate the amount of storage in the replication log comprise instructions configured to cause the at least one processor to
allocate storage in the replication log for the each data volume based on the scheduling priority of the each data volume.
22. The computer-implemented method of claim 1 , further comprising:
receiving one or more write operation requests at a node of the plurality of nodes, wherein
the node is configured to receive the one or more write operation requests from one or more of the plurality of nodes;
scheduling one or more write operations corresponding to the one or more write operation requests by adding the one or more write operation requests to a global queue;
reading the one or more write operation requests from the global queue; and
performing the one or more write operations in a first-in, first-out (FIFO) order, wherein
the cluster comprises shared storage, and
each of the one or more write operations is performed by executing the each of the one or more write operations on a replication log and subsequently on the shared storage.