Dynamic slice assignment in a distributed storage system
An apparatus comprises at least one processing device that includes a processor coupled to a memory, and is configured to monitor latencies associated with processing of input-output operations in a plurality of storage nodes of a distributed storage system, to detect an unbalanced condition between the storage nodes based at least in part on the monitored latencies, and responsive to the detected unbalanced condition, to adjust an assignment of slices of a logical address space of the distributed storage system to the storage nodes. Adjusting the assignment of slices of the logical address space of the distributed storage system to the storage nodes responsive to the detected unbalanced condition illustratively comprises increasing a number of the slices assigned to one or more of the storage nodes having relatively low latencies and decreasing a number of slices assigned to one or more of the storage nodes having relatively high latencies.
1. An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
said at least one processing device being configured:
to monitor latencies associated with processing of input-output operations in a plurality of storage nodes of a distributed storage system;
to detect an unbalanced condition between the storage nodes based at least in part on the monitored latencies; and
responsive to the detected unbalanced condition, to adjust an assignment of slices of a logical address space of the distributed storage system to the storage nodes;
wherein the slices of the logical address space comprise respective sets of contiguous pages corresponding to respective different fractional portions of a given logical storage volume;
wherein the different fractional portions of the given logical storage volume are stored on different ones of the storage nodes utilizing the corresponding slices assigned to that storage node;
wherein each of at least a subset of a plurality of storage nodes of the distributed storage system comprises a set of processing modules configured to communicate with corresponding sets of processing modules on other ones of the storage nodes, the sets of processing modules of the storage nodes of the distributed storage system collectively comprising at least a portion of a distributed storage controller of the distributed storage system;
wherein monitoring latencies associated with processing of input-output operations in the storage nodes comprises:
obtaining latency measurements from flow control components implemented in respective ones of the processing modules implemented in respective ones of the storage nodes; and
utilizing the latency measurements in a feedback loop for adjusting a choker size to limit concurrent processing of input-output operations in the distributed storage system;
wherein one or more of the sets of processing modules each comprise at least a routing module, a control module and a data module; and
wherein adjusting an assignment of slices of the logical address space of the distributed storage system to the storage nodes responsive to the detected unbalanced condition comprises modifying a number of the slices assigned to control modules of one or more of the storage nodes based at least in part on latencies of the corresponding one or more storage nodes.
2. The apparatus of claim 1 wherein said at least one processing device comprises at least one processing core of a multi-core storage node of the distributed storage system.
3. The apparatus of claim 1 wherein monitoring latencies associated with processing of input-output operations in the storage nodes comprises:
periodically obtaining latency measurements from flow control components implemented in respective ones of the storage nodes; and
computing individual latencies of the respective ones of the storage nodes and an average latency of the respective storage nodes based at least in part on the obtained latency measurements;
the flow control components being part of an adaptive flow control mechanism of the distributed storage system;
the adaptive flow control mechanism comprising a feedback loop in which a window size defining an amount of concurrent processing of the input-output operations in the distributed storage system is adjusted responsive to a measured latency in processing of one or more of the input-output operations;
wherein the measured latency for processing of one or more of the input-output operations comprises an average end-to-end latency measured over multiple ones of the input-output operations; and
wherein the window size defines a maximum permitted amount of concurrent processing of the input-output operations in the distributed storage system in terms of a number of data units of a particular size.
4. The apparatus of claim 1 wherein detecting an unbalanced condition between the storage nodes based at least in part on the monitored latencies comprises determining if at least one of the storage nodes has a relatively high latency that deviates by more than a threshold amount from an average latency of the storage nodes.
5. The apparatus of claim 1 wherein adjusting an assignment of slices of the logical address space of the distributed storage system to the storage nodes responsive to the detected unbalanced condition comprises increasing a number of the slices assigned to one or more of the storage nodes having relatively low latencies and decreasing a number of slices assigned to one or more of the storage nodes having relatively high latencies.
6. The apparatus of claim 1 wherein adjusting an assignment of slices of the logical address space of the distributed storage system to the storage nodes responsive to the detected unbalanced condition comprises modifying an initial assignment of the slices to the storage nodes in which the slices are evenly distributed across the storage nodes with each of the storage nodes being assigned the same number of slices.
7. The apparatus of claim 1 wherein adjusting an assignment of slices of the logical address space of the distributed storage system to the storage nodes responsive to the detected unbalanced condition comprises increasing a number of the slices assigned to control modules of one or more of the storage nodes having relatively low latencies and decreasing a number of slices assigned to control modules of one or more of the storage nodes having relatively high latencies.
8. The apparatus of claim 1 wherein adjusting an assignment of slices of the logical address space of the distributed storage system to the storage nodes responsive to the detected unbalanced condition comprises modifying an initial assignment of the slices to the control modules of the storage nodes in which the slices are evenly distributed across the control modules of the storage nodes with each of the control modules of the storage nodes being assigned the same number of slices.
9. The apparatus of claim 1 wherein monitoring latencies associated with processing of input-output operations in the storage nodes comprises periodically obtaining latency measurements from flow control components implemented in respective ones of the routing modules implemented in respective ones of the storage nodes.
10. The apparatus of claim 9 wherein a given one of the flow control components in a corresponding one of the routing modules is configured:
to determine latency of its corresponding routing module;
to determine latencies of each of at least a subset of the control modules;
to determine latencies of each of at least a subset of the data modules; and
to report the determined latencies to a system manager module implemented in at least one of the storage nodes, the system manager module processing the latencies received from the given flow control component in the corresponding one of the routing modules and latencies received from other ones of the flow control components in other ones of the routing modules to compute individual latencies of respective ones of the storage nodes and an average latency of the respective storage nodes.
11. The apparatus of claim 10 wherein the system manager module computes an individual latency of a given one of the storage nodes by summing an average routing module latency, an average control module latency and an average data module latency for the given storage node over a particular measurement interval.
12. The apparatus of claim 1 wherein adjusting an assignment of slices of the logical address space of the distributed storage system to the storage nodes responsive to the detected unbalanced condition comprises:
making a first adjustment by increasing a number of the slices assigned to one or more of the storage nodes having relatively low latencies and decreasing a number of slices assigned to one or more of the storage nodes having relatively high latencies;
determining whether or not the unbalanced condition still exists based at least in part on continued monitoring of latencies subsequent to making the first adjustment; and
responsive to the unbalanced condition still existing, making an additional adjustment by further increasing the number of the slices assigned to one or more of the storage nodes having relatively low latencies and further decreasing a number of slices assigned to one or more of the storage nodes having relatively high latencies;
wherein one or more additional iterations of the determining and if necessary one or more further iterations of making an additional adjustment are performed until it is determined that the unbalanced condition no longer exists.
13. The apparatus of claim 1 wherein detecting an unbalanced condition between the storage nodes based at least in part on the monitored latencies comprises detecting an unbalanced condition attributable to at least one of:
failure and restart of at least one processing module of at least one of the storage nodes; and
at least one storage drive failure impacting at least one of the storage nodes.
14. The apparatus of claim 1 wherein adjusting the assignment of slices to the storage nodes results in a first one of the different fractional portions of the given logical storage volume being reassigned from a first one of the storage nodes to a second one of the storage nodes while a second one of the different fractional portions of the given logical storage volume remains assigned to the first storage node.
15. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:
to monitor latencies associated with processing of input-output operations in a plurality of storage nodes of a distributed storage system;
to detect an unbalanced condition between the storage nodes based at least in part on the monitored latencies; and
responsive to the detected unbalanced condition, to adjust an assignment of slices of a logical address space of the distributed storage system to the storage nodes;
wherein the slices of the logical address space comprise respective sets of contiguous pages corresponding to respective different fractional portions of a given logical storage volume;
wherein the different fractional portions of the given logical storage volume are stored on different ones of the storage nodes utilizing the corresponding slices assigned to that storage node;
wherein each of at least a subset of a plurality of storage nodes of the distributed storage system comprises a set of processing modules configured to communicate with corresponding sets of processing modules on other ones of the storage nodes, the sets of processing modules of the storage nodes of the distributed storage system collectively comprising at least a portion of a distributed storage controller of the distributed storage system;
wherein monitoring latencies associated with processing of input-output operations in the storage nodes comprises:
obtaining latency measurements from flow control components implemented in respective ones of the processing modules implemented in respective ones of the storage nodes; and
utilizing the latency measurements in a feedback loop for adjusting a choker size to limit concurrent processing of input-output operations in the distributed storage system;
wherein one or more of the sets of processing modules each comprise at least a routing module, a control module and a data module; and
wherein adjusting an assignment of slices of the logical address space of the distributed storage system to the storage nodes responsive to the detected unbalanced condition comprises modifying a number of the slices assigned to control modules of one or more of the storage nodes based at least in part on latencies of the corresponding one or more storage nodes.
16. The computer program product of claim 15 wherein detecting an unbalanced condition between the storage nodes based at least in part on the monitored latencies comprises determining if at least one of the storage nodes has a relatively high latency that deviates by more than a threshold amount from an average latency of the storage nodes.
17. The computer program product of claim 15 wherein adjusting an assignment of slices of the logical address space of the distributed storage system to the storage nodes responsive to the detected unbalanced condition comprises increasing a number of the slices assigned to one or more of the storage nodes having relatively low latencies and decreasing a number of slices assigned to one or more of the storage nodes having relatively high latencies.
18. A method comprising:
monitoring latencies associated with processing of input-output operations in a plurality of storage nodes of a distributed storage system;
detecting an unbalanced condition between the storage nodes based at least in part on the monitored latencies; and
responsive to the detected unbalanced condition, adjusting an assignment of slices of a logical address space of the distributed storage system to the storage nodes;
wherein the slices of the logical address space comprise respective sets of contiguous pages corresponding to respective different fractional portions of a given logical storage volume;
wherein the different fractional portions of the given logical storage volume are stored on different ones of the storage nodes utilizing the corresponding slices assigned to that storage node;
wherein each of at least a subset of a plurality of storage nodes of the distributed storage system comprises a set of processing modules configured to communicate with corresponding sets of processing modules on other ones of the storage nodes, the sets of processing modules of the storage nodes of the distributed storage system collectively comprising at least a portion of a distributed storage controller of the distributed storage system; and
wherein monitoring latencies associated with processing of input-output operations in the storage nodes comprises:
obtaining latency measurements from flow control components implemented in respective ones of the processing modules implemented in respective ones of the storage nodes; and
utilizing the latency measurements in a feedback loop for adjusting a choker size to limit concurrent processing of input-output operations in the distributed storage system;
wherein one or more of the sets of processing modules each comprise at least a routing module, a control module and a data module;
wherein adjusting an assignment of slices of the logical address space of the distributed storage system to the storage nodes responsive to the detected unbalanced condition comprises modifying a number of the slices assigned to control modules of one or more of the storage nodes based at least in part on latencies of the corresponding one or more storage nodes; and
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.
19. The method of claim 18 wherein detecting an unbalanced condition between the storage nodes based at least in part on the monitored latencies comprises determining if at least one of the storage nodes has a relatively high latency that deviates by more than a threshold amount from an average latency of the storage nodes.
20. The method of claim 18 wherein adjusting an assignment of slices of the logical address space of the distributed storage system to the storage nodes responsive to the detected unbalanced condition comprises increasing a number of the slices assigned to one or more of the storage nodes having relatively low latencies and decreasing a number of slices assigned to one or more of the storage nodes having relatively high latencies.
21. The method of claim 18 wherein adjusting the assignment of slices to the storage nodes results in a first one of the different fractional portions of the given logical storage volume being reassigned from a first one of the storage nodes to a second one of the storage nodes while a second one of the different fractional portions of the given logical storage volume remains assigned to the first storage node.