Provider-agnostic management of backup data stored on a cloud-based object storage
Described is a system for managing backup data stored on a cloud-based object storage irrespective of the storage provider. The system may include a server (or gateway) that provides a backup service, and acts as an intermediary between a client device and object storage. To provide such capabilities, the system may manage an application programming interface (API) that provides a uniform set of methods to a client device irrespective of the destination object storage provider. A function library may include provider-specific functions that invoke corresponding storage-layer operations provided by the destination object storage. The server may also manage an operation pool to schedule operations and saturate bandwidth to the object storage. The server may also implement a specialized metadata database to provide further storage efficiencies by storing certain backup data exclusively within the metadata database.
1. A system comprising:
one or more processors; and
a non-transitory computer-readable medium storing a plurality of instructions, which when executed, cause the one or more processors to:
initialize, by a server, a set of components including at least an operation pool configured to queue operations to be performed on a first type of object storage or a second type of object storage that are cloud-based;
forward a request, received from a first client device, to access the first type of object storage, the request including a credential authorizing the first client device to access the first type of object storage;
provide, to the first client device, an identifier associated with the first client device, in response to the first type of object storage verifying the credential;
receive, from the first client device, a write request to write client data to the first type of object storage, the write request including an identification of the client data, and the identifier;
select, based on the identifier and amongst a set of a first write function that is specific to the first type of object storage or a second write function that is specific to the second type of object storage, the first write function that is specific to the first type of object storage; and
add a write object to the operation pool, the write object configured to execute the first write function to write client data that is specific to the first type of object storage.
2. The system of claim 1 , wherein the write request to write client data to the first type of object storage is received via an application programming interface (API) associated with the server.
3. The system of claim 1 , wherein the first write function is configured to use an API associated with the first type of object storage to write the client data to the first type of object storage.
4. The system of claim 1 , wherein adding the write object to the operation pool includes adding the write object to a write operation queue of the operation pool, the write object further configured to execute the write function as a set of sub-operations.
5. The system of claim 4 , wherein the operation pool is initialized based on one or more configuration settings received from the first client device.
6. The system of claim 5 , wherein the one or more configuration settings include a number of write sub-operations that may be performed concurrently on the first type of object storage.
7. The system of claim 1 , wherein the set of components further includes a metadata database configured to store metadata associated with the client data.
8. The system of claim 7 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
determine the write request includes changes to the metadata associated with the client data; and
update the metadata database to store the changed metadata, in response to receiving the write request.
9. A method comprising:
initializing, by a server, a set of components including at least an operation pool configured to queue operations to be performed on a first type of object storage or a second type of object storage that are cloud-based;
forwarding a request, received from a first client device, to access the first type of object storage, the request including a credential authorizing the first client device to access the first type of object storage;
providing, to the first client device, an identifier associated with the first client device, in response to the first type of object storage verifying the credential;
receiving, from the first client device, a write request to write client data to the first type of object storage, the write request including an identification of the client data, and the identifier;
selecting, based on the identifier and amongst a set of a first write function that is specific to the first type of object storage or a second write function that is specific to the second type of object storage, the first write function that is specific to the first type of object storage; and
adding a write object to the operation pool, the write object configured to execute the first write function to write the client data that is specific to the first type of object storage.
10. The method of claim 9 , wherein the write request to write client data to the first type of object storage is received via an application programming interface (API) associated with the server.
11. The method of claim 9 , wherein the first write function is configured to use an API associated with the first type of object storage to write the client data to the first type of object storage.
12. The method of claim 9 , wherein adding the write object to the operation pool includes adding the write object to a write operation queue of the operation pool, the write object further configured to execute the write function as a set of sub-operations.
13. The method of claim 9 , wherein the set of components further includes a metadata database configured to store metadata associated with the client data.
14. The method of claim 13 , further comprising:
determining the write request includes changes to the metadata associated with the client data; and
updating the metadata database to store the changed metadata, in response to receiving the write request.
15. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
initialize, by a server, a set of components including at least an operation pool configured to queue operations to be performed on a first type of object storage or a second type of object storage that are cloud-based;
forward a request, received from a first client device, to access the first type of object storage, the request including a credential authorizing the first client device to access the first type of object storage;
provide, to the first client device, an identifier associated with the first client device, in response to the first type of object storage verifying the credential;
receive, from the first client device, a write request to write client data to the first type of object storage, the write request including an identification of the client data, and the identifier;
select, based on the identifier and amongst a set of a first write function that is specific to a first type of object storage or a second write function that is specific to a second type of object storage, the first write function that is specific to the first type of object storage; and
add a write object to the operation pool, the write object configured to execute the first write function to write the client data that is specific to the first type of object storage.
16. The computer program product of claim 15 , wherein the write request to write client data to the first type of object storage is received via an application programming interface (API) associated with the server.
17. The computer program product of claim 15 , wherein the first write function is configured to use an API associated with the first type of object storage to write the client data to the first type of object storage.
18. The computer program product of claim 15 , wherein adding the write object to the operation pool includes adding the write object to a write operation queue of the operation pool, the write object further configured to execute the write function as a set of sub-operations.
19. The computer program product of claim 15 , wherein the set of components further includes a metadata database configured to store metadata associated with the client data.
20. The computer program product of claim 19 , wherein the program code includes further instructions to:
determine the write request includes changes to the metadata associated with the client data; and
update the metadata database to store the changed metadata, in response to receiving the write request.