Cloud storage brokering service
Systems and methods for brokering storage services are disclosed. In one embodiment, the brokering system is configured to receive, from a client device, a request indicating a desired configuration for storing target data in at least one of the plurality of cloud storage services. The system can determine a first selection comprising at least one storage service from the plurality of cloud storage services based on the request, provision storage in the plurality of cloud storage services based on the determined first selection and provide, to the client device, a temporary manifest indicating the provisioned storage, wherein the temporary manifest comprises access data that enables the client device to access the provisioned storage in the cloud storage services. The system can then store the temporary manifest and determine whether the client device completed storing the target data on the provisioned storage based on a presence of the temporary manifest.
1. A method of enabling upload of a file to a cloud storage system, said method comprising:
receiving, from a client device, a request to upload a file, wherein the request indicates a size of the file;
creating a storage container at at least one storage service of a plurality of storage services;
receiving, from the at least one storage service, a set of access tokens for the storage container set;
generating a temporary storage manifest including the access tokens, a data set identifier, and an address of the storage container;
transmitting, to the client device, the temporary storage manifest;
receiving a notification from the client device that the file was uploaded to the storage container successfully, the notification including the data set identifier; and
in response to the notification, deleting the temporary storage manifest.
2. The method of claim 1 , further comprising establishing a customer account with the at least one storage service for uploading the file.
3. The method of claim 2 , further comprising maintaining a file manifest mapping the file to the customer account.
4. The method of claim 3 , wherein:
portions of the file are stored at a plurality of the plurality of storage services; and
the file manifest comprises cloud platform specific details for accessing each of the portions of the file using the plurality of the plurality of storage services.
5. The method of claim 1 , further comprising:
automatically arbitrating with the plurality of storage services; and
automatically initiating migration of at least a portion of the file between the plurality of storage services.
6. The method of claim 5 , further comprising maintaining a file manifest for the file for the at least one storage service.
7. A method of enabling download of a file from a cloud storage system, said method comprising:
receiving, from a client device, a request to download a file, wherein the request comprises a unique identifier for the file;
locating a storage manifest that indicates respective locations of blocks of the file in at least one storage service based on the unique identifier;
retrieving, from the at least one storage service, access tokens for the blocks of the file;
transmitting, to the client device, the storage manifest with the retrieved access tokens for downloading of the file;
receiving a notification from the client device that downloading of the file is complete; and
in response to the notification, revoking the access tokens.
8. The method of claim 7 , further comprising:
automatically arbitrating with a plurality of storage services; and
automatically initiating migration of at least some of the blocks of the file between the plurality of storage services.
9. The method of claim 8 , further comprising maintaining the storage manifest for the file for the at least one storage service.
10. A system for enabling upload of a file to a cloud storage system, the system comprising:
a memory; and
at least one processor coupled to the memory and configured to:
receive, from a client device, a request to upload a file, wherein the request indicates a size of the file;
create a storage container at at least one storage service of a plurality of storage services;
receive, from the at least one storage service, access tokens for the storage container;
generate a temporary storage manifest including the access tokens, a data set identifier, and an address of the storage container;
transmit, to the client device, the temporary storage manifest;
receive a notification from the client device that the file was uploaded to the storage container successfully, the notification including the data set identifier; and
delete the temporary manifest in response to the notification.
11. The system of claim 10 , wherein the at least one processor is further configured to establish a customer account with the at least one storage service for uploading the file.
12. The system of claim 11 , wherein the at least one processor is further configured to maintain a file manifest mapping the file to the customer account.
13. The system of claim 12 , wherein:
portions of the file are stored at a plurality of storage services; and
the file manifest comprises cloud platform specific details for accessing each of the portions of the file using the plurality of storage services.
14. The system of claim 10 , wherein the at least one processor is further configured to:
automatically arbitrate with the plurality of storage services; and
automatically initiate migration of at least a portion of the file between the plurality of storage services.
15. The system of claim 14 , wherein the at least one processor is further configured to maintain a file manifest for the file for the at least one storage service.
16. A system for enabling download of a file from a cloud storage system, the system comprising:
a memory; and
at least one processor coupled to the memory and configured to:
receive, from a client device, a request to download a file, wherein the request comprises a unique identifier for the file;
locate a manifest that indicates respective locations of blocks of the file in at least one storage service based on the unique identifier;
retrieve, from the at least one storage service, access tokens for the blocks of the file;
transmit, to the client device, the manifest with the retrieved access tokens for downloading of the file;
receive a notification from the client device that downloading of the file is complete; and
in response to the notification, revoke the access tokens.
17. The system of claim 16 , wherein the at least one processor is further configured to:
automatically arbitrate with a plurality of storage services; and
automatically initiate migration of at least some of the blocks of the file between the plurality of storage services.
18. The system of claim 17 , wherein the at least one processor is further configured to maintain the manifest for the file.