I/O scheduling and load balancing across the multiple nodes of a clustered environment using a single global queue
View Patent ↗Write operations are scheduled for multiple nodes in a shared storage cluster that supports volume replication. Requests are received from nodes for allocation of space for write operations in a replication log. In response to a received request, the current capacity of the requesting node to manage a backlog can be determined. The amount of space in the replication log allocated to the node is then calibrated to the node's capacity, thereby preventing self-throttling. A separate priority can be assigned to each volume, and space in the replication log assigned to each volume based on its priority. Nodes can target synchronous and other latency sensitive operations to higher priority volumes. A single global queue can be maintained to schedule write operations for all nodes, thereby providing a fair scheduling. A separate local queue can be maintained for each node, thereby providing specific levels of preference to specific nodes.
1. A computer implemented method for scheduling write operations of a plurality of nodes, thereby balancing a load of storage input/output operations in a clustering and storage environment that utilizes shared storage and supports replication of data volumes, the method comprising the steps of:
maintaining, by a computer, a single global queue to schedule write operations for each of the plurality of nodes of the clustering and storage environment, wherein the each node adds requests to schedule the write operations to the single global queue;
reading, by the computer, the requests to schedule the write operations from the single global queue; and
scheduling the write operations from the single global queue in first in first out order, wherein the scheduled write operations are written first to a replication log and subsequently to the shared storage, wherein an order of write operations to the replication log matches an order of write operations to the shared storage, and wherein the clustering and storage environment utilizes the order and content of the write operations to the replication log to replicate the data volumes of the shared storage to a secondary site, thereby preserving write-order fidelity.
2. A computer implemented method for scheduling write operations of a plurality of nodes, thereby balancing a load of storage input/output operations in a clustering and storage environment that utilizes shared storage and supports replication of data volumes, the method comprising the steps of:
maintaining, by a computer, a separate local write operation scheduling queue for each one of the plurality of nodes of the clustering and storage environment, wherein the each node adds requests to schedule write operations to its separate local write operation scheduling queue;
reading, by the computer, the requests to schedule the write operations from each separate local write operation scheduling queue in first in first out order; and
scheduling, by the computer, the write operations corresponding to the requests read from the each separate local write operation scheduling queue so as to provide specific levels of scheduling preference to specific ones of the plurality of nodes, wherein the scheduled write operations are written first to a replication log and subsequently to the shared storage, wherein an order of write operations to the replication log matches an order of write operations to the shared storage, and wherein the clustering and storage environment utilizes the order and content of the write operations to the replication log to replicate the data volumes of the shared storage to a secondary site, thereby preserving write-order fidelity.
3. The method of claim 2 wherein the scheduling the write operations corresponding to the requests read from the each separate local write operation scheduling queue so as to provide specific levels of scheduling preference to the specific ones of the plurality of nodes further comprises:
providing equal scheduling preference to the each node, by allocating a substantially equal amount of space in the replication log to the each node.
4. The method of claim 2 wherein the scheduling the write operations corresponding to the requests read from the each separate local write operation scheduling queue so as to provide specific levels of scheduling preference to the specific ones of the plurality of nodes further comprises:
providing equal scheduling preference to the each node, by periodically dividing available space in the replication log substantially equally between the each node.
5. The method of claim 2 wherein the scheduling the write operations corresponding to the requests read from the each separate local write operation scheduling queue so as to provide specific levels of scheduling preference to the specific ones of the plurality of nodes further comprises:
providing a level of scheduling preference to each specific node in direct proportion to a number of requests for write operations in the specific node's separate local write operation scheduling queue, by allocating an amount space in the replication log to the each specific node in direct proportion to the specific node's number of queued requests.
6. The method of claim 2 wherein the scheduling the write operations corresponding to the requests read from the each separate local write operation scheduling queue so as to provide specific levels of scheduling preference to the specific ones of the plurality of nodes further comprises:
receiving input entered by a user indicating a level of scheduling preference to provide to a specific node; and
providing the level of scheduling preference indicated by the received input to the specific node.
7. The method of claim 2 wherein the scheduling the write operations corresponding to the requests read from the each separate local write operation scheduling queue so as to provide specific levels of scheduling preference to the specific ones of the plurality of nodes further comprises:
receiving input entered by a user indicating an adjustment to make to a level of scheduling preference to provide to a specific node; and
adjusting the level of scheduling preference provided to the specific node as indicated by the received input.
8. The method of claim 2 further comprising:
receiving the requests from the nodes of the clustering and storage environment for allocation of space for the write operations, each one of the received requests further comprises information indicative of a priority of a corresponding write operation; and
scheduling the write operations at a level of priority indicated in corresponding received requests.
9. A non-transitory computer readable-storage medium for scheduling write operations of a plurality of nodes, thereby balancing a load of storage input/output operations in a clustering and storage environment that utilizes shared storage and supports replication of data volumes, the non-transitory computer readable-storage medium storing computer executable instructions that, when loaded into computer memory and executed by at least one processor of a computing device, cause the computing device to perform the following steps:
maintaining a separate local write operation scheduling queue for each one of the plurality of nodes of the clustering and storage environment, wherein the each node adds requests to schedule write operations to its separate local write operation scheduling queue;
reading the requests to schedule the write operations from the each separate local write operation scheduling queue in first in first out order; and
scheduling the write operations corresponding to the requests read from the each separate local write operation scheduling queue so as to provide specific levels of scheduling preference to specific ones of the plurality of nodes, wherein the scheduled write operations are written first to a replication log and subsequently to the shared storage, wherein an order of write operations to the replication log matches an order of write operations to the shared storage, and wherein the clustering and storage environment utilizes the order and content of the write operations to the replication log to replicate the data volumes of the shared storage to a secondary site, thereby preserving write-order fidelity.
10. The non-transitory computer readable-storage medium of claim 9 wherein the scheduling the write operations corresponding to the requests read from the each separate local write operation scheduling queue so as to provide specific levels of scheduling preference to the specific ones of the plurality of nodes further comprises:
providing equal scheduling preference to the each node, by allocating a substantially equal amount of space in the replication log to the each node.
11. The non-transitory computer readable-storage medium of claim 9 wherein the scheduling the write operations corresponding to the requests read from the each separate local write operation scheduling queue so as to provide specific levels of scheduling preference to the specific ones of the plurality of nodes further comprises:
providing equal scheduling preference to the each node, by periodically dividing available space in the replication log substantially equally between the each node.
12. The non-transitory computer readable-storage medium of claim 9 wherein the scheduling the write operations corresponding to the requests read from the each separate local write operation scheduling queue so as to provide specific levels of scheduling preference to the specific ones of the plurality of nodes further comprises:
providing a level of scheduling preference to each specific node in direct proportion to a number of requests for write operations in the specific node's separate local write operation scheduling queue, by allocating an amount space in the replication log to the each specific node in direct proportion to the specific node's number of queued requests.
13. The non-transitory computer readable-storage medium of claim 9 wherein the scheduling the write operations corresponding to the requests read from the each separate local write operation scheduling queue so as to provide specific levels of scheduling preference to the specific ones of the plurality of nodes further comprises:
receiving input entered by a user indicating a level of scheduling preference to provide to a specific node; and
providing the level of scheduling preference indicated by the received input to the specific node.
14. The non-transitory computer readable-storage medium of claim 9 wherein the scheduling the write operations corresponding to the requests read from the each separate local write operation scheduling queue so as to provide specific levels of scheduling preference to the specific ones of the plurality of nodes further comprises:
receiving input entered by a user indicating an adjustment to make to a level of scheduling preference to provide to a specific node; and
adjusting the level of scheduling preference provided to the specific node as indicated by the received input.
15. The non-transitory computer readable-storage medium of claim 9 further comprising:
receiving the requests from the nodes of the clustering and storage environment for allocation of space for the write operations, each one of the received requests further comprises information indicative of a priority of a corresponding write operation; and
scheduling the write operations at a level of priority indicated in corresponding received requests.