Bandwidth management in distributed file systems
Embodiments are directed to bandwidth management in distributed file systems. A request stored in a queue may be dequeued. A work credit value may be determined based on a duration since a last job and a work rate associated with the file system such that the work credit value conforms to a maximum threshold value. A job may be initialized based on the dequeued request such that the work credit value may be reduced based on a size of the job. In response to the reduced work credit value being a negative value further actions may be performed, including: determining a time delay based on the reduced work credit value, the work rate, and the size of the job; delaying an execution of the job until the expiry of the time delay such that upon expiry of the time delay, the job is executed to satisfy the request.
1. A method for managing data in a file system over a network using one or more processors that execute instructions to enable performance of actions, comprising:
dequeuing a request stored in a queue based on availability of computational resources;
determining a duration based on a first timestamp that corresponds to the dequeuing of the request and a second timestamp that corresponds to a prior dequeuing of a previous request;
determining a work credit value based on the duration and a work rate associated with the file system, wherein the work credit value conforms to a maximum threshold value;
initializing a job based on the dequeued request, wherein the work credit value is reduced based on a size of the job; and
in response to the reduced work credit value being a negative value, performing further actions, including:
determining a time delay based on the reduced work credit value, the work rate, and the size of the job; and
delaying an execution of the job until the expiry of the time delay, wherein upon expiry of the time delay, the job is executed to satisfy the request.
2. The method of claim 1 , further comprising:
determining two or more adjacent requests in the queue, wherein each adjacent request corresponds to a same file system object and an adjacent data location in the file system object;
determining a portion of the two or more adjacent requests that have a combined data size that is less than a threshold value; and
coalescing the portion of the two or more adjacent requests into a single request in the queue.
3. The method of claim 1 , further comprising:
in response to the dequeued request being directed to reading or writing metadata corresponding to one or more file system objects, executing the job immediately.
4. The method of claim 1 , further comprising:
updating the second timestamp to match the first timestamp.
5. The method of claim 1 , further comprising:
storing the request in the queue based on a type associated with the request; and
determining the queue from a plurality of queues, wherein each queue corresponds to a data write request type, a data read request type, a metadata write request type, or a metadata read request type.
6. The method of claim 1 , further comprising:
dequeuing another request from the queue or another queue; and
determining another work credit value based on another duration, another work rate associated with the file system, and the other work credit value.
7. The method of claim 1 , further comprising:
determining the queue based on a type of the request; and
storing the request in the determined queue.
8. A system for managing data in a file system comprising:
a network computer, comprising:
a memory that stores at least instructions; and
one or more processors that execute instructions that perform actions, including:
dequeuing a request stored in a queue based on availability of computational resources;
determining a duration based on a first timestamp that corresponds to the dequeuing of the request and a second timestamp that corresponds to a prior dequeuing of a previous request;
determining a work credit value based on the duration and a work rate associated with the file system, wherein the work credit value conforms to a maximum threshold value;
initializing a job based on the dequeued request, wherein the work credit value is reduced based on a size of the job; and
in response to the reduced work credit value being a negative value, performing further actions, including:
determining a time delay based on the reduced work credit value, the work rate, and the size of the job; and
delaying an execution of the job until the expiry of the time delay, wherein upon expiry of the time delay, the job is executed to satisfy the request; and
one or more client computers, comprising:
a memory that stores at least instructions; and
one or more processors that execute instructions that perform actions, including, providing the request to the file system.
9. The system of claim 8 , wherein the one or more network computer processors execute instructions that perform actions, further comprising:
determining two or more adjacent requests in the queue, wherein each adjacent request corresponds to a same file system object and an adjacent data location in the file system object;
determining a portion of the two or more adjacent requests that have a combined data size that is less than a threshold value; and
coalescing the portion of the two or more adjacent requests into a single request in the queue.
10. The system of claim 8 , wherein the one or more network computer processors execute instructions that perform actions, further comprising:
in response to the dequeued request being directed to reading or writing metadata corresponding to one or more file system objects, executing the job immediately.
11. The system of claim 8 , wherein the one or more network computer processors execute instructions that perform actions, further comprising:
updating the second timestamp to match the first timestamp.
12. The system of claim 8 , wherein the one or more network computer processors execute instructions that perform actions, further comprising:
storing the request in the queue based on a type associated with the request; and
determining the queue from a plurality of queues, wherein each queue corresponds to a data write request type, a data read request type, a metadata write request type, or a metadata read request type.
13. The system of claim 8 , wherein the one or more network computer processors execute instructions that perform actions, further comprising:
dequeuing another request from the queue or another queue; and
determining another work credit value based on another duration, another work rate associated with the file system, and the other work credit value.
14. The system of claim 8 , wherein the one or more network computer processors execute instructions that perform actions, further comprising:
determining the queue based on a type of the request; and
storing the request in the determined queue.
15. A processor readable non-transitory storage media that includes instructions for managing data in a file system over a network, wherein execution of the instructions by one or more processors on one or more network computers performs actions, comprising:
dequeuing a request stored in a queue based on availability of computational resources;
determining a duration based on a first timestamp that corresponds to the dequeuing of the request and a second timestamp that corresponds to a prior dequeuing of a previous request;
determining a work credit value based on the duration and a work rate associated with the file system, wherein the work credit value conforms to a maximum threshold value;
initializing a job based on the dequeued request, wherein the work credit value is reduced based on a size of the job; and
in response to the reduced work credit value being a negative value, performing further actions, including:
determining a time delay based on the reduced work credit value, the work rate, and the size of the job; and
delaying an execution of the job until the expiry of the time delay, wherein upon expiry of the time delay, the job is executed to satisfy the request.
16. The media of claim 15 , further comprising:
determining two or more adjacent requests in the queue, wherein each adjacent request corresponds to a same file system object and an adjacent data location in the file system object;
determining a portion of the two or more adjacent requests that have a combined data size that is less than a threshold value; and
coalescing the portion of the two or more adjacent requests into a single request in the queue.
17. The media of claim 15 , further comprising:
in response to the dequeued request being directed to reading or writing metadata corresponding to one or more file system objects, executing the job immediately.
18. The media of claim 15 , further comprising:
updating the second timestamp to match the first timestamp.
19. The media of claim 15 , further comprising:
storing the request in the queue based on a type associated with the request; and
determining the queue from a plurality of queues, wherein each queue corresponds to a data write request type, a data read request type, a metadata write request type, or a metadata read request type.
20. The media of claim 15 , further comprising:
dequeuing another request from the queue or another queue; and
determining another work credit value based on another duration, another work rate associated with the file system, and the other work credit value.
21. The media of claim 15 , further comprising:
determining the queue based on a type of the request; and
storing the request in the determined queue.
22. A network computer for managing data in a file system, comprising:
a memory that stores at least instructions; and
one or more processors that execute instructions that perform actions, including:
dequeuing a request stored in a queue based on availability of computational resources;
determining a duration based on a first timestamp that corresponds to the dequeuing of the request and a second timestamp that corresponds to a prior dequeuing of a previous request;
determining a work credit value based on the duration and a work rate associated with the file system, wherein the work credit value conforms to a maximum threshold value;
initializing a job based on the dequeued request, wherein the work credit value is reduced based on a size of the job; and
in response to the reduced work credit value being a negative value, performing further actions, including:
determining a time delay based on the reduced work credit value, the work rate, and the size of the job; and
delaying an execution of the job until the expiry of the time delay, wherein upon expiry of the time delay, the job is executed to satisfy the request.
23. The network computer of claim 22 , wherein the one or more processors execute instructions that perform actions, further comprising:
determining two or more adjacent requests in the queue, wherein each adjacent request corresponds to a same file system object and an adjacent data location in the file system object;
determining a portion of the two or more adjacent requests that have a combined data size that is less than a threshold value; and
coalescing the portion of the two or more adjacent requests into a single request in the queue.
24. The network computer of claim 22 , wherein the one or more processors execute instructions that perform actions, further comprising:
in response to the dequeued request being directed to reading or writing metadata corresponding to one or more file system objects, executing the job immediately.
25. The network computer of claim 22 , wherein the one or more processors execute instructions that perform actions, further comprising:
updating the second timestamp to match the first timestamp.
26. The network computer of claim 22 , wherein the one or more processors execute instructions that perform actions, further comprising:
storing the request in the queue based on a type associated with the request; and
determining the queue from a plurality of queues, wherein each queue corresponds to a data write request type, a data read request type, a metadata write request type, or a metadata read request type.
27. The network computer of claim 22 , wherein the one or more processors execute instructions that perform actions, further comprising:
dequeuing another request from the queue or another queue; and
determining another work credit value based on another duration, another work rate associated with the file system, and the other work credit value.
28. The network computer of claim 22 , wherein the one or more processors execute instructions that perform actions, further comprising:
determining the queue based on a type of the request; and
storing the request in the determined queue.