Cloud-based scheduling of data operations to an object storage using an operation pool
Described is a system for cloud-based scheduling of data operations to an object storage. The system may include a server that acts as an intermediary between a client device and an object storage that stores client data. Requests by the client device to manage data on the object storage may be routed through the server. In response, the server may execute corresponding data operations directly on the object storage. The server may manage bandwidth limits for operation requests to an object storage by maintaining a specialized operation pool. The operation pool may be configured to manage operation-specific queues, and these operation-specific queues may include sub-operations (or threads) that are executed on the object storage. Sub-operations of various types of operations may be initiated for concurrent execution on the object storage based on scheduling information derived from the operation pool.
1. A system comprising:
one or more processors; and
a non-transitory computer readable medium storing a plurality of instructions, which when executed, cause the one or more processors to:
create, by a server that manages communication between a client device and an object storage, a communication session with the object storage, the communication session allowing access to one or more types of data operations for managing data stored on the object storage;
initialize an operation pool for scheduling data operations on the object storage, the operation pool storing an operation queue for each of the types of data operations, each operation queue storing one or more operation objects that each represent an operation to be performed on the object storage, and each operation object identifying a set of sub-operations of a corresponding operation to be performed on the object storage;
receive, at the server and from the client device, a request to perform a first data operation on the object storage;
add a first operation object representing the first data operation to a first operation queue associated with the first data operation, the first operation object identifying a first set of sub-operations of the first data operation; and
initiate, by the server, one or more of the first set of sub-operations based on a number of operation requests available to the object storage and the operation pool.
2. The system of claim 1 , wherein the number of operation requests available to the object storage includes a current number of concurrent operation requests available to the object storage, and the one or more of the first set of sub-operations are initiated concurrently.
3. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
divide the first data operation into the first set of sub-operations based on a resource requirement associated with the first operation.
4. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
determine the first data operation is a first type of data operation; and
identify the first operation queue as being associated with the first type of data operation.
5. The system of claim 1 , wherein initiating one or more of the first set of sub-operations includes accessing a first method from an application programming interface (API) provided by the object storage, the first method corresponding to the first data operation.
6. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
receive, after the request to perform the first data operation and from the client device, a request to perform a second data operation on the object storage;
add a second operation object representing the second data operation to a second operation queue associated with the second data operation, the second operation object identifying a second set of sub-operations of the second data operation; and
initiate, by the server, one or more of the second set of sub-operations based on a number of operation requests available to the object storage after initiating the first set of sub-operations.
7. The system of claim 6 , wherein one of the first data operation and second data operation is a write operation, and one of the first data operation and second data operation is a read operation.
8. The system of claim 1 , wherein the number of operation requests available to the object storage includes a current number of concurrent first data operation requests available to the object storage.
9. A method comprising:
creating, by a server that manages communication between a client device and an object storage, a communication session with the object storage, the communication session allowing access to one or more types of data operations for managing data stored on the object storage;
initializing an operation pool for scheduling data operations on the object storage, the operation pool storing an operation queue for each of the types of data operations, each operation queue storing one or more operation objects that each represent an operation to be performed on the object storage, and each operation object identifying a set of sub-operations of a corresponding operation to be performed on the object storage;
receiving, at the server and from the client device, a request to perform a first data operation on the object storage;
adding a first operation object representing the first data operation to a first operation queue associated with the first data operation, the first operation object identifying a first set of sub-operations of the first data operation; and
initiating, by the server, one or more of the first set of sub-operations based on a number of operation requests available to the object storage and the operation pool.
10. The method of claim 9 , wherein the number of operation requests available to the object storage includes a current number of concurrent operation requests available to the object storage, and the one or more of the first set of sub-operations are initiated concurrently.
11. The method of claim 9 , further comprising:
dividing the first operation into the first set of sub-operations based on a resource requirement associated with the first operation.
12. The method of claim 9 , further comprising:
determining the first data operation is a first type of data operation; and
identifying the first operation queue as being associated with the first type of data operation.
13. The method of claim 9 , wherein initiating one or more of the first set of sub-operations includes accessing a first method from an application programming interface (API) provided by the object storage, the first method corresponding to the first data operation.
14. The method of claim 9 , further comprising:
receiving, after the request to perform the first data operation and from the client device, a request to perform a second data operation on the object storage;
adding a second operation object representing the second data operation to a second operation queue associated with the second data operation, the second operation object identifying a second set of sub-operations of the second data operation; and
initiating, by the server, one or more of the second set of sub-operations based on a number of operation requests available to the object storage after initiating the first set of sub-operations.
15. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
create, by a server that manages communication between a client device and an object storage, a communication session with the object storage, the communication session allowing access to one or more types of data operations for managing data stored on the object storage;
initialize an operation pool for scheduling data operations on the object storage, the operation pool storing an operation queue for each of the types of data operations, each operation queue storing one or more operation objects that each represent an operation to be performed on the object storage, and each operation object identifying a set of sub-operations of a corresponding operation to be performed on the object storage;
receive, at the server and from the client device, a request to perform a first data operation on the object storage;
add a first operation object representing the first data operation to a first operation queue associated with the first data operation, the first operation object identifying a first set of sub-operations of the first data operation; and
initiate, by the server, one or more of the first set of sub-operations based on a number of operation requests available to the object storage and the operation pool.
16. The computer program product of claim 15 , wherein the number of operation requests available to the object storage includes a current number of concurrent operation requests available to the object storage, and the one or more of the first set of sub-operations are initiated concurrently.
17. The computer program product of claim 15 , wherein the program code includes further instructions to:
divide the first operation into the first set of sub-operations based on a resource requirement associated with the first operation.
18. The computer program product of claim 15 , wherein the program code includes further instructions to:
determine the first data operation is a first type of data operation; and
identify the first operation queue as being associated with the first type of data operation.
19. The computer program product of claim 15 , wherein initiating one or more of the first set of sub-operations includes accessing a first method from an application programming interface (API) provided by the object storage, the first method corresponding to the first data operation.
20. The computer program product of claim 15 , wherein the program code includes further instructions to:
receive, after the request to perform the first data operation and from the client device, a request to perform a second data operation on the object storage;
add a second operation object representing the second data operation to a second operation queue associated with the second data operation, the second operation object identifying a second set of sub-operations of the second data operation; and
initiate, by the server, one or more of the second set of sub-operations based on a number of operation requests available to the object storage after initiating the first set of sub-operations.