I/O scheduling and load balancing across the multiple nodes of a clustered environment
Write operations are scheduled for multiple nodes in a shared storage cluster that supports volume replication. Requests are received from nodes for allocation of space for write operations in a replication log. In response to a received request, the current capacity of the requesting node to manage a backlog can be determined. The amount of space in the replication log allocated to the node is then calibrated to the node's capacity, thereby preventing self-throttling. A separate priority can be assigned to each volume, and space in the replication log assigned to each volume based on its priority. Nodes can target synchronous and other latency sensitive operations to higher priority volumes. A single global queue can be maintained to schedule write operations for all nodes, thereby providing a fair scheduling. A separate local queue can be maintained for each node, thereby providing specific levels of preference to specific nodes.
1. A computer implemented method for scheduling write operations of a plurality of nodes, thereby balancing a load of storage input/output operations in a clustering and storage environment that utilizes shared storage and supports replication of data volumes, the method comprising the steps of:
receiving, by a computer, requests from nodes of the clustering and storage environment for allocation of space for write operations in a replication log, wherein nodes perform write operations by writing first to the replication log and subsequently to the shared storage, wherein an order of write operations to the replication log matches an order of write operations to the shared storage, and wherein the clustering and storage environment utilizes the order and content of write operations to the replication log to replicate the data volumes of the shared storage to a secondary site, thereby preserving write-order fidelity;
responsive to receiving a request from a specific node for the allocation of space in the replication log, determining, by the computer, a current capacity of the specific node to manage a backlog of write operations without throttling execution of its own write operations; and
calibrating, by the computer, an amount of space in the replication log to allocate to the determined current capacity of the specific node.
2. The method of claim 1 wherein calibrating an amount of space in the replication log to allocate to the determined current capacity of the specific node further comprises:
allocating the specific node only enough space in the replication log to execute a number of backlogged write operations that the specific node can simultaneously manage without throttling its own write operations.
3. The method of claim 1 wherein calibrating an amount of space in the replication log to allocate to the determined current capacity of the specific node further comprises:
limiting allocation of space in the replication log to an upper limit of the specific node for simultaneous backlogged write operation management, thereby preventing the specific node from throttling its backlogged write operations above the upper limit.
4. The method of claim 1 wherein determining a current capacity of a specific node to manage a backlog of write operations without throttling execution of its own write operations further comprises:
gleaning information indicative of the current capacity of the specific node from a request received from the specific node for allocation of space for write operations in the replication log, wherein each received request for allocation of space in the replication log further comprises information indicative of the current capacity of the requesting node.
5. The method of claim 4 wherein the information indicative of the current capacity of the requesting node in received requests further comprises:
a number of outstanding storage input/out requests pending on the node, and the node's Small Computer System Interface (“SCSI”) tag command queue.
6. The method of claim 1 wherein receiving requests from nodes of the clustering and storage environment for allocation of space for write operations further comprises:
receiving requests from nodes of the clustering and storage environment for allocation of space for write operations, each one of said received requests further comprising information indicative of a priority of the corresponding write operation; and
scheduling write operations at a level of priority indicated in corresponding received requests.
7. The method of claim 1 wherein determining a current capacity of a specific node to manage a backlog of write operations without throttling execution of its own write operations further comprises:
periodically polling each one of the plurality of nodes for information indicative of the current capacity of the specific node.