Using machine learning techniques to flow control clients in a deduplication file system
Detecting a trend in latency for storage underlying a deduplicated file system includes generating a set of data points by recording when input/output (IO) requests were issued to the storage and recording time required to receive success responses from the storage. Least squares regression is performed on the data points to find a best-fit line through the data points. A slope of the best-fit line is calculated. A determination is made as to whether the slope is positive, a positive slope thereby indicating a trend of increasing latency of the storage. When the slope is determined to be positive, clients accessing the deduplicated file system are throttled.
1. A method comprising:
detecting a trend in latency for storage underlying a deduplicated file system comprising:
generating a plurality of data points by recording over a sampling interval timestamps indicating when input/output (IO) requests were issued by the deduplicated file system to the storage, and recording time required to receive success responses from the storage;
performing a least squares regression on the plurality of data points to find a best-fit line through the data points;
calculating a slope of the best-fit line; and
determining whether the slope is positive, a positive slope thereby indicating a trend of increasing latency of the storage; and
when the slope is determined to be positive, throttling clients that are accessing the deduplicated file system.
2. The method of claim 1 wherein the generating a plurality of data points further comprises:
dividing the sampling interval into a plurality of intermediate time periods of equal duration;
identifying IO requests issued by the deduplicated file system to the storage during an intermediate time period, and, for each IO request, time required to receive the success responses from the storage;
calculating an average time to receive the success responses for the IO requests issued during the intermediate time period; and
pairing the intermediate time period with the average time required to receive the success responses for the IO requests issued during the intermediate time period of the sampling interval to form an xy-coordinate of a data point.
3. The method of claim 1 wherein the throttling comprises:
generating a notification supported by a communication protocol used by the clients to inform the clients that their requests could not be processed and that the clients should wait and retry; and
transmitting the notification to the clients.
4. The method of claim 1 wherein the throttling comprises:
advertising to the clients a memory pool of the deduplicated file system as having zero capacity.
5. The method of claim 1 further comprising:
continuing the throttling of the clients until the slope of the best-fit line is determined to be zero or negative.
6. The method of claim 1 further comprising:
before the detecting a trend in latency for the storage, monitoring utilization of a queue of the deduplicated file system holding tasks to be performed by the deduplicated file system; and
when a capacity of the queue reaches a threshold percentage, initiating detection of the trend in latency for the storage.
7. A system comprising: a processor; and memory configured to store one or more sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of:
detecting a trend in latency for storage underlying a deduplicated file system comprising:
generating a plurality of data points by recording over a sampling interval timestamps indicating when input/output (IO) requests were issued by the deduplicated file system to the storage, and recording time required to receive success responses from the storage;
performing a least squares regression on the plurality of data points to find a best-fit line through the data points;
calculating a slope of the best-fit line; and
determining whether the slope is positive, a positive slope thereby indicating a trend of increasing latency of the storage; and
when the slope is determined to be positive, throttling clients that are accessing the deduplicated file system.
8. The system of claim 7 wherein the generating a plurality of data points further comprises:
dividing the sampling interval into a plurality of intermediate time periods of equal duration;
identifying IO requests issued by the deduplicated file system to the storage during an intermediate time period, and, for each IO request, time required to receive the success responses from the storage;
calculating an average time to receive the success responses for the IO requests issued during the intermediate time period; and
pairing the intermediate time period with the average time required to receive the success responses for the IO requests issued during the intermediate time period of the sampling interval to form an xy-coordinate of a data point.
9. The system of claim 7 wherein the throttling comprises:
generating a notification supported by a communication protocol used by the clients to inform the clients that their requests could not be processed and that the clients should wait and retry; and
transmitting the notification to the clients.
10. The system of claim 7 wherein the throttling comprises:
advertising to the clients a memory pool of the deduplicated file system as having zero capacity.
11. The system of claim 7 wherein the processor further carries out the step of:
continuing the throttling of the clients until the slope of the best-fit line is determined to be zero or negative.
12. The system of claim 7 wherein the processor further carries out the steps of:
before the detecting a trend in latency for the storage, monitoring utilization of a queue of the deduplicated file system holding tasks to be performed by the deduplicated file system; and
when a capacity of the queue reaches a threshold percentage, initiating detection of the trend in latency for the storage.
13. A computer program product, comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method comprising:
detecting a trend in latency for storage underlying a deduplicated file system comprising:
generating a plurality of data points by recording over a sampling interval timestamps indicating when input/output (IO) requests were issued by the deduplicated file system to the storage, and recording time required to receive success responses from the storage;
performing a least squares regression on the plurality of data points to find a best-fit line through the data points;
calculating a slope of the best-fit line; and
determining whether the slope is positive, a positive slope thereby indicating a trend of increasing latency of the storage; and
when the slope is determined to be positive, throttling clients that are accessing the deduplicated file system.
14. The computer program product of claim 13 wherein the generating a plurality of data points further comprises:
dividing the sampling interval into a plurality of intermediate time periods of equal duration;
identifying IO requests issued by the deduplicated file system to the storage during an intermediate time period, and, for each IO request, time required to receive the success responses from the storage;
calculating an average time to receive the success responses for the IO requests issued during the intermediate time period; and
pairing the intermediate time period with the average time required to receive the success responses for the IO requests issued during the intermediate time period of the sampling interval to form an xy-coordinate of a data point.
15. The computer program product of claim 13 wherein the throttling comprises:
generating a notification supported by a communication protocol used by the clients to inform the clients that their requests could not be processed and that the clients should wait and retry; and
transmitting the notification to the clients.
16. The computer program product of claim 13 wherein the throttling comprises:
advertising to the clients a memory pool of the deduplicated file system as having zero capacity.
17. The computer program product of claim 13 wherein the method further comprises:
continuing the throttling of the clients until the slope of the best-fit line is determined to be zero or negative.
18. The computer program product of claim 13 wherein the method further comprises:
before the detecting a trend in latency for the storage, monitoring utilization of a queue of the deduplicated file system holding tasks to be performed by the deduplicated file system; and
when a capacity of the queue reaches a threshold percentage, initiating detection of the trend in latency for the storage.