Token-based data flow control in a clustered storage system
An apparatus in one embodiment comprises a storage system including multiple storage nodes each having a plurality of storage devices. Each of the storage nodes further comprises a set of processing modules configured to communicate with corresponding sets of processing modules on other ones of the storage nodes. The sets of processing modules of the storage nodes collectively comprise at least a portion of a distributed storage controller of the storage system. The distributed storage controller is configured to implement token-based data flow control between designated ones of the modules by determining a token distribution for the designated modules, the token distribution providing an allocation of tokens to particular ones of the designated modules. A given one of the modules is configured to limit its generation of messages to other modules based at least in part on availability of corresponding ones of the tokens allocated to the given module.
1. An apparatus comprising:
a storage system comprising a plurality of storage nodes each having a plurality of storage devices;
each of the storage nodes further comprising a set of processing modules configured to communicate over one or more networks with corresponding sets of processing modules on other ones of the storage nodes;
the sets of processing modules of the storage nodes collectively comprising at least a portion of a distributed storage controller of the storage system;
wherein the distributed storage controller is configured to implement token-based data flow control between designated ones of the modules by determining a token distribution for the designated modules, the token distribution providing an allocation of tokens to particular ones of the designated modules;
wherein a given one of the modules is configured to limit its generation of messages to other ones of the modules based at least in part on availability of corresponding ones of the tokens allocated to the given module;
wherein the designated modules comprise control modules of respective ones of the storage nodes and wherein the token-based data flow control limits numbers of control-to-routing messages generated by the control modules to each of a plurality of routing modules of respective ones of the storage nodes based at least in part on availability of tokens to the control modules;
wherein the token distribution for the designated modules varies over time as a function of measures relating to data transfer rates, message processing rates and average message latencies of respective corresponding communication links associated with the designated modules; and
wherein each of the storage nodes is implemented using at least one processing device comprising a processor coupled to a memory.
2. The apparatus of claim 1 wherein the distributed storage controller is further configured to collect statistics characterizing communications between the modules and to adjust the token distribution dynamically over time based at least in part on the collected statistics.
3. The apparatus of claim 2 wherein the distributed storage controller is further configured to adjust the token distribution dynamically over time based at least in part on the collected statistics by increasing a number of allocated tokens for at least one of the control modules responsive to an actual latency being less than a desired latency and decreasing a number of allocated tokens for at least one of the control modules responsive to the actual latency being greater than a desired latency.
4. The apparatus of claim 1 wherein the set of processing modules of each of the storage nodes comprises at least a control module, a data module and a routing module.
5. The apparatus of claim 1 wherein the token distribution is implemented at least in part utilizing a token distribution table indicating for each of the control modules a particular number of tokens allocated for use in conjunction with generation of the control-to-routing messages by that control module to each of the routing modules.
6. The apparatus of claim 1 wherein the token distribution specifies a particular number of tokens for each of a plurality of communication links between a given one of the control modules and respective ones of the routing modules.
7. The apparatus of claim 1 wherein a given one of the control modules prior to sending one of the control-to-routing messages to one of the routing modules first determines if a token is available for its communication link to that routing module and if the token is available transmits the control-to-routing message and if the token is not available does not transmit the control-to-routing message.
8. The apparatus of claim 7 wherein after the given one of the control modules transmits the control-to-routing message to the routing module a corresponding one of the allocated tokens becomes unavailable until a response to the control-to-routing message is received from the routing module.
9. The apparatus of claim 1 wherein the distributed storage controller is further configured to determine initial numbers of tokens to be allocated to respective ones of the designated modules based at least in part on ratios of numbers of data slices associated with respective ones of the designated modules to a total number of data slices in the storage system.
10. The apparatus of claim 1 wherein the distributed storage controller is further configured to transmit information characterizing at least portions of the token distribution to control modules in the sets of processing modules of the respective storage nodes.
11. The apparatus of claim 1 wherein the distributed storage controller is further configured to alter the token distribution responsive to a change in a distribution of data slices over the processing modules.
12. The apparatus of claim 1 wherein the token-based data flow control is implemented by at least one system-wide management module of the distributed storage controller.
13. A method comprising:
configuring a storage system to include a plurality of storage nodes each having a plurality of storage devices, each of the storage nodes further comprising a set of processing modules configured to communicate over one or more networks with corresponding sets of processing modules on other ones of the storage nodes; and
implementing token-based data flow control between designated ones of the modules by determining a token distribution for the designated modules, the token distribution providing an allocation of tokens to particular ones of the designated modules;
wherein a given one of the modules is configured to limit its generation of messages to other ones of the modules based at least in part on availability of corresponding ones of the tokens allocated to the given module;
wherein the designated modules comprise control modules of respective ones of the storage nodes and wherein the token-based data flow control limits numbers of control-to-routing messages generated by the control modules to each of a plurality of routing modules of respective ones of the storage nodes based at least in part on availability of tokens to the control modules;
wherein the token distribution for the designated modules varies over time as a function of measures relating to data transfer rates, message processing rates and average message latencies of respective corresponding communication links associated with the designated modules; and
wherein the method is implemented by at least one processing device comprising a processor coupled to a memory.
14. The method of claim 13 further comprising:
collecting statistics characterizing communications between the modules; and
adjusting the token distribution dynamically over time based at least in part on the collected statistics.
15. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:
to configure a storage system to include a plurality of storage nodes each having a plurality of storage devices, each of the storage nodes further comprising a set of processing modules configured to communicate over one or more networks with corresponding sets of processing modules on other ones of the storage nodes; and
to implement token-based data flow control between designated ones of the modules by determining a token distribution for the designated modules, the token distribution providing an allocation of tokens to particular ones of the designated modules;
wherein a given one of the modules is configured to limit its generation of messages to other ones of the modules based at least in part on availability of corresponding ones of the tokens allocated to the given module;
wherein the designated modules comprise control modules of respective ones of the storage nodes and wherein the token-based data flow control limits numbers of control-to-routing messages generated by the control modules to each of a plurality of routing modules of respective ones of the storage nodes based at least in part on availability of tokens to the control modules; and
wherein the token distribution for the designated modules varies over time as a function of measures relating to data transfer rates, message processing rates and average message latencies of respective corresponding communication links associated with the designated modules.
16. The computer program product of claim 15 wherein the program code when executed by said at least one processing device further causes said at least one processing device:
to collect statistics characterizing communications between the modules; and
to adjust the token distribution dynamically over time based at least in part on the collected statistics.
17. The computer program product of claim 15 wherein the token distribution is implemented at least in part utilizing a token distribution table indicating for each of the control modules a particular number of tokens allocated for use in conjunction with generation of the control-to-routing messages by that control module to each of the routing modules.
18. The computer program product of claim 15 wherein the token distribution specifies a particular number of tokens for each of a plurality of communication links between a given one of the control modules and respective ones of the routing modules.
19. The computer program product of claim 15 wherein a given one of the control modules prior to sending one of the control-to-routing messages to one of the routing modules first determines if a token is available for its communication link to that routing module and if the token is available transmits the control-to-routing message and if the token is not available does not transmit the control-to-routing message.
20. The computer program product of claim 15 wherein the distributed storage controller is further configured to determine initial numbers of tokens to be allocated to respective ones of the designated modules based at least in part on ratios of numbers of data slices associated with respective ones of the designated modules to a total number of data slices in the storage system.