Congestion mitigation in a distributed storage system
A system comprises a plurality of computing devices that are communicatively coupled via a network and have a file system distributed among them, and comprises one or more file system request buffers residing on one or more of the plurality of computing devices. File system choking management circuitry that resides on one or more of the plurality of computing devices is operable to separately control: a first rate at which a first type of file system requests (e.g., one of data requests, data read requests, data write requests, metadata requests, metadata read requests, and metadata write requests) are fetched from the one or more buffers, and a second rate at which a second type of file system requests (e.g., another of data requests, data read requests, data write requests, metadata requests, metadata read requests, and metadata write requests) are fetched from the one or more buffers.
1 . A distributed storage system (DSS) comprising:
a circuit configured to:
control, according to a batch size, a first rate at which a first type of requests are fetched from a request buffer; and
control a second rate at which a second type of requests are fetched from the request buffer, wherein:
the circuit is operable to control the first rate according to a batch timing, and
the circuit is operable to adjust a batch timing setting and a batch size setting.
2 . A distributed storage system (DSS) comprising:
a circuit configured to:
control, according to a batch size, a first rate at which a first type of requests are fetched from a request buffer; and
control a second rate at which a second type of requests are fetched from the request buffer, wherein:
the first type of requests comprise requests to read data from the DSS and requests to write data to the DSS; and
the second type of requests comprise requests to read metadata from the DSS and requests to write metadata to the DSS.
3 . A distributed storage system (DSS) comprising:
a circuit configured to:
control, according to a batch size, a first rate at which a first type of requests are fetched from a request buffer; and
control a second rate at which a second type of requests are fetched from the request buffer, wherein the circuit is operable to adjust:
a data batch timing setting;
a data batch size setting;
a metadata batch timing setting; and
a metadata batch size setting.
4 . A distributed storage system (DSS) comprising:
a circuit configured to:
control, according to a batch size, a first rate at which a first type of requests are fetched from a request buffer; and
control a second rate at which a second type of requests are fetched from the request buffer, wherein:
the first type of requests comprise requests to read data from the DSS and requests to read metadata from the DSS; and
the second type of requests comprise requests to write data to the DSS and requests to write metadata to the DSS.
5 . A distributed storage system (DSS) comprising:
a circuit configured to:
control, according to a batch size, a first rate at which a first type of requests are fetched from a request buffer; and
control a second rate at which a second type of requests are fetched from the request buffer, wherein the circuit is operable to adjust:
a read batch timing setting;
a read batch size setting;
a write batch timing setting; and
a write batch size setting.
6 . A distributed storage system (DSS) comprising:
a circuit configured to:
control, according to a batch size, a first rate at which a first type of requests are fetched from a request buffer; and
control a second rate at which a second type of requests are fetched from the request buffer, wherein the circuit is operable to control, independently from the first rate and the second rate, a third rate at which a third type of requests are fetched from one or more buffers.
7 . A distributed storage system (DSS) comprising:
a circuit configured to:
control, according to a batch size, a first rate at which a first type of requests are fetched from a request buffer; and
control a second rate at which a second type of requests are fetched from the request buffer, wherein the circuit is operable to adjust:
a data read batch timing setting;
a data read batch size setting;
a data write batch timing setting;
a data write batch size setting;
a metadata batch timing setting; and
a metadata batch size setting.
8 . A distributed storage system (DSS) comprising:
a circuit configured to:
control, according to a batch size, a first rate at which a first type of requests are fetched from a request buffer; and
control a second rate at which a second type of requests are fetched from the request buffer, wherein the circuit is operable to control according to a current load on one or more resources of a computing device and according to a predicted load on one or more resources of a plurality of computing devices.
9 . The DSS of claim 8 , wherein:
the one or more resources comprise a storage device; and
a load on one or more resources of the plurality of computing devices is determined according to a depth of a buffer of the storage device.
10 . A distributed storage system (DSS) comprising:
a circuit configured to:
control, according to a batch size, a first rate at which a first type of requests are fetched from a request buffer; and
control a second rate at which a second type of requests are fetched from the request buffer, wherein:
the circuit is operable to calculate a choking level according to a current load and a predicted load on one or more resources of a plurality of computing devices; and
the control of the first rate and of the second rate is determined according to the choking level.
11 . The DSS of claim 10 , wherein the calculation of the choking level comprises a mapping, according to a first determined function, of a plurality of resource load values for resources of the plurality of computing devices to a plurality of congestion contribution values.
12 . The DSS of claim 11 , wherein the plurality of resource load values comprise two or more of:
a network load value;
a processing core load value;
a memory load value; and
a storage device load value.
13 . The DSS of claim 12 , wherein the storage device load value is determined according to a depth of a buffer of a storage device of the plurality of computing devices.
14 . The DSS of claim 11 , wherein the plurality of resource load values comprises a storage device read load value and a storage device write load value.
15 . The DSS of claim 11 wherein:
the resources of the plurality of computing devices comprise a storage device that stores data and/or metadata of the DSS; and
the first determined function is determined based on characteristics of the storage device.
16 . The DSS of claim 15 , wherein the characteristics of the storage device are determined according to one or more of:
a manufacturer of the storage device;
a manufacturer of a component of the storage device;
a model of the storage device;
a model of a component of the storage device;
a serial number of the storage device; and
a serial number of a component of the storage device.
17 . The DSS of claim 15 , wherein the characteristics of the storage device comprise input/output operations per second (IOPS).
18 . The DSS of claim 10 , wherein the calculation of the choking level comprises a combining of the plurality of congestion contribution values according to a second determined function.
19 . A distributed storage system (DSS) comprising:
a circuit configured to:
control, according to a batch size, a first rate at which a first type of requests are fetched from a request buffer; and
control a second rate at which a second type of requests are fetched from the request buffer, wherein:
the circuit is configured to provide an application programming interface via one or more settings used by the circuit for the control, and
the first rate and the second rate are configurable during runtime of the circuit.
20 . A distributed storage system (DSS) comprising:
a circuit configured to:
control, according to a batch size, a first rate at which a first type of requests are fetched from a request buffer; and
control a second rate at which a second type of requests are fetched from the request buffer, wherein the circuit is operable to:
determine characteristics of hardware of a plurality of computing devices;
perform an initial configuration of one or more settings used by the circuit for the control the first rate and the second rate according to the characteristics of the hardware; and
adapt the one or more settings as requests that are serviced and/or queued for servicing by the DSS.