Quality of service management in a distributed storage system
One or more computing devices may comprise congestion management circuitry, one or more client file system request buffers, and DESS interface circuitry. The one or more client file system request buffers is/are operable to queue first client file system requests of a first priority level and second client file system requests of a second priority level, where the first priority level is higher priority than the second priority level. The DESS interface circuitry is operable to determine a choking level according to the load on a plurality of DESS resources. Individual load values of the DESS resources are mapped to a composite load value using a first function. The composite load value is mapped to a congestion contribution using a second function. And, the congestion contribution is mapped to a choking level using a third function.
1. A system comprising:
a plurality of buffers comprising a plurality of file system requests; and
queueing circuitry operable to determine a current choking level of the plurality of buffers, wherein:
a first buffer, of the plurality of buffers, is operable to queue file system requests, of the plurality of file system requests, associated with a first priority level,
a second buffer, of the plurality of buffers, is operable to queue file system requests, of the plurality of file system requests, associated with a second priority level that is lower than the first priority level,
the current choking level is determined according to a normalization of the plurality of file system requests and a plurality of individual loads on a plurality of respective resources, wherein:
the normalization is relative to a number of input/output operations (IOPs), and
the number of IOPs is dynamically updated according to a machine learning algorithm,
the plurality of individual loads are associated with a composite load, and
the composite load is associated with a congestion.
2. The system of claim 1 , wherein the queueing circuitry is operable to determine a priority of the plurality of file system requests according to a mount point to which each file system request is directed.
3. The system of claim 1 , wherein the queueing circuitry is operable to queue a file system request in a buffer of the plurality of buffers according to a mount point to which the file system request is directed.
4. The system of claim 3 , wherein a file system is mounted at two or more mount points.
5. The system of claim 1 , wherein the queueing circuitry is operable to update the current choking level according to a previous choking level associated with file system requests previously queued in the plurality of buffers.
6. The system of claim 1 , wherein the queueing circuitry is operable to fetch file system requests from the plurality of buffers in accordance with a round-robin schedule and one or more choking levels.
7. The system of claim 6 , wherein the round-robin schedule is such that, in each round of the round-robin:
at least one file system request directed to a file system mounted at a first mount point is fetched; and
at least one file system request directed to a file system mounted at a second mount point is fetched.
8. The system of claim 1 , comprising management circuitry operable to determine, based on characteristics of a node, where to mount a file system.
9. The system of claim 8 , wherein the characteristics of the node comprise a process that runs on the node.
10. The system of claim 1 , wherein the queueing circuitry is operable to:
fetch file system requests from the first buffer and from the second buffer when the congestion is below a threshold; and
fetch file system requests from the first buffer but not from the second buffer when the congestion is above the threshold.
11. A method comprising:
buffering a plurality of file system requests in a plurality of buffers;
queuing, via a first buffer of the plurality of buffers, file system requests, of the plurality of file system requests, associated with a first priority level;
queuing, via a second buffer of the plurality of buffers, file system requests, of the plurality of file system requests, associated with a second priority level that is lower than the first priority level;
normalizing the plurality of file system requests relative to a number of input/output operations (IOPs), wherein the number of IOPs is dynamically updated according to a machine learning algorithm;
determining a plurality of individual loads on a plurality of respective resources; and
determining a current choking level of the plurality of buffers according to the normalization of the plurality of file system requests and the plurality of individual loads on a plurality of respective resources, wherein:
the plurality of individual loads are associated with a composite load, and
the composite load is associated with a congestion.
12. The method of claim 11 , wherein the method comprises:
determining a priority of the plurality of file system requests according to a mount point to which each file system request is directed.
13. The method of claim 11 , wherein the method comprises:
queuing a file system request in a buffer of the plurality of buffers according to a mount point to which the file system request is directed.
14. The method of claim 13 , wherein a file system is mounted at two or more mount points.
15. The method of claim 11 , wherein the method comprises:
updating the current choking level according to a previous choking level associated with file system requests previously queued in the plurality of buffers.
16. The method of claim 11 , wherein the method comprises:
fetching file system requests from the plurality of buffers in accordance with a round-robin schedule and one or more choking levels.
17. The method of claim 16 , wherein the round-robin schedule is such that, in each round of the round-robin:
at least one file system request directed to a file system mounted at a first mount point is fetched; and
at least one file system request directed to a file system mounted at a second mount point is fetched.
18. The method of claim 11 , wherein the method comprises:
determining, based on characteristics of a node, where to mount a file system.
19. The method of claim 18 , wherein the characteristics of the node comprise a process that runs on the node.
20. The method of claim 11 , wherein the method comprises:
fetching file system requests from the first buffer and from the second buffer when the congestion is below a threshold; and
fetching file system requests from the first buffer but not from the second buffer when the congestion is above the threshold.