System and method for optimized input/output to an object storage system
Disclosed are systems, methods and computer readable mediums for optimized throughput of an object based storage system. The systems, methods and computer readable mediums including receiving an I/O request to the storage system, determining a busy ratio based on a number of blocks available in a local cache and a queue size, determining an I/O speed to the storage system, the I/O speed based at least in part on the busy ratio and an upload speed, wherein the I/O speed does not exceed a current speed of the storage system, and executing the I/O request to the storage system at the I/O speed.
1. A method for performing I/O on a storage system, the method comprising:
receiving, at a device, an I/O request to the storage system using a first protocol;
translating the I/O request to a second protocol, wherein the second protocol is specific to the device;
queuing the I/O request in a local block cache;
determining a busy ratio by:
determining, via a top layer of an object backer of the storage system, a number of blocks available in the local block cache;
querying, via the top layer of the object backer, a plurality of background threads running in the storage system, and determining, based on the query, a number of independent threads identified from the background threads as currently managing write calls to the storage system;
calculating a maximum queue size for the local block cache, the calculating comprising multiplying the number of blocks by the number of independent threads identified from the background threads as currently managing write calls to the storage system; and
calculating the busy ratio based on a current size of the queue and the calculated maximum queue size for the local block cache;
determining an I/O speed from the device to the storage system, the I/O speed based at least in part on the busy ratio and an upload speed, wherein the I/O speed does not exceed a current speed of the storage system; and
executing, at the device, the I/O request to the storage system at the I/O speed.
2. The method of claim 1 , wherein the second protocol is one of AWS S3, S3 compatible, Azure Blob, or Swift.
3. The method of claim 1 , further comprising translating a second I/O request from the first protocol to a third protocol.
4. The method of claim 1 , wherein the translation is performed by a statically linked device driver translator.
5. The method of claim 1 , wherein the translation is performed by a dynamically linked device driver translator.
6. The method of claim 1 , further comprising identifying the second protocol based on one of an IP address and a DNS lookup of the storage system.
7. The method of claim 1 , further comprising determining a number of independent threads, wherein the independent threads perform I/O operations between the device and an object storage system.
8. The method of claim 7 , further comprising:
determining an upload speed for each independent thread; and
synchronizing executions of I/O operations across the independent threads.
9. The method of claim 1 , wherein the object storage system is a cloud-based storage system.
10. The method of claim 1 , wherein the busy ratio is calculated by dividing the queue size by a maximum queue size.
11. The method of claim 1 , wherein the upload speed is determined by multiplying a number of blocks sent by a block size and dividing by an elapsed time.
12. The method of claim 1 , wherein the upload speed is limited based on a percentage of the busy ratio.
13. The method of claim 12 , wherein the upload speed is inversely proportional to the percentage value of the busy ratio.
14. A system for performing I/O on a storage system, the system comprising:
one or more processors; and
a memory, the memory comprising instructions which, when executed by the one or more processors, cause the one or more processors to:
receive an I/O request to the storage system using a first protocol;
translate the I/O request to a second protocol, wherein the second protocol is specific to the device;
queue the I/O request in a local block cache;
determine a busy ratio by:
determining, via a top layer of an object backer of the storage system, a number of blocks available in the local block cache;
querying, via the top layer of the object backer, a plurality of background threads running in the storage system, and determining, based on the query, a number of independent threads identified from the background threads as currently managing write calls to the storage system;
calculating a maximum queue size for the local block cache, the calculating comprising multiplying the number of blocks by the number of independent threads identified from the background threads as currently managing write calls to the storage system; and
calculating the busy ratio based on a current size of the queue and the calculated maximum queue size for the local block cache;
determine an I/O speed to the storage system, the I/O speed based at least in part on the busy ratio and an upload speed, wherein the I/O speed does not exceed a current speed of the storage system; and
execute the I/O request to the storage system at the I/O speed.
15. The system of claim 14 , wherein the second protocol is one of AWS S3, S3 compatible, Azure Blob, or Swift.
16. A non-transitory computer readable medium storing instructions, which when executed by one or more processors, causes the one or more processors to:
receive an I/O request to the storage system using a first protocol;
translate the I/O request to a second protocol, wherein the second protocol is specific to the device;
queue the I/O request in a local block cache;
determine a busy ratio by:
determining, via a top layer of an object backer of the storage system, a number of blocks available in the local block cache;
querying, via the top layer of the object backer, a plurality of background threads running in the storage system, and determining, based on the query, a number of independent threads identified from the background threads as currently managing write calls to the storage system;
calculating a maximum queue size for the local block cache, the calculating comprising multiplying the number of blocks by the number of independent threads identified from the background threads as currently managing write calls to the storage system; and
calculating the busy ratio based on a current size of the queue and the calculated maximum queue size for the local block cache;
determine an I/O speed to the storage system, the I/O speed based at least in part on the busy ratio and an upload speed, wherein the I/O speed does not exceed a current speed of the storage system; and
execute the I/O request to the storage system at the I/O speed.