Efficient transfer to and from a deduplicated cloud storage system
Efficient data transfer is disclosed. A server provides an endpoint for a client. The client can communicate with a cloud storage system through the server rather than connect directly to the cloud storage system. The server is configured to perform requests in a manner that reduces the amount of data transferred to and from the cloud storage system.
1. A method for transferring objects to and from a cloud storage system, the method comprising:
receiving a request from a client at a server with regard to an object, wherein the client is configured to direct the request to an endpoint of the server;
determining a request type of the request by the server;
when the request type is a PUT request, performing the following data transfer operations, wherein the data transfer operations are configured to reduce an amount of data transferred between the server and the cloud storage system:
preparing, by the server, an object for transmission by dividing the object into slices and dividing the slices into segments;
communicating, by the server, with the cloud storage system to identify which segments of the object not stored at the cloud object storage;
transferring only the segments of the object that are not stored at the cloud object storage to the cloud storage system, wherein the cloud storage system incorporates the missing segments into the cloud object storage; and
storing the in accordance with a protocol of the cloud storage system.
2. The method of claim 1 , wherein the request type is one of a PUT request, a GET request, and an OTHER request.
3. The method of claim 2 , when the request type is the GET request, further comprising:
communicating, by the server, with the cloud storage system to request an object, wherein the object is divided into slices and the slices are divided into segments;
receiving compressed slices from the cloud storage system at the server;
decompressing, at the server, the compressed slices;
stitching the decompressed slices together to reconstruct the object; and
returning the reconstructed object to the client.
4. The method of claim 3 , further comprising decompressing the compressed slices while streaming the compressed slices without landing the compressed slices at the server.
5. The method of claim 4 , further comprising reconstructing the decompressed slices at the server while streaming to the client.
6. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
a request from a client at a server with regard to an object, wherein the client is configured to direct the request to an endpoint of the server;
determining a request type of the request by the server; operations, wherein the data transfer operations are configured to reduce an amount of data transferred between the server and the cloud storage system:
preparing, by the server, an object for transmission by dividing the object into slices and dividing the slices into segments;
which segments of the object not stored at the cloud object storage;
transferring only the segments of the object that are not stored at the cloud object storage to the cloud storage system, wherein the cloud storage system incorporates the missing segments into the cloud object storage; and
storing the in accordance with a protocol of the cloud storage system.
7. The non-transitory storage medium of claim 6 , wherein the request type is one of a PUT request, a GET request, and an OTHER request.
8. The non-transitory storage medium of claim 6 , further comprising generating a fingerprint for each segment included in the segments.
9. The non-transitory storage medium of claim 7 , when the request type is the GET request, further comprising: communicating, by the server, with the cloud storage system to request an object, wherein the object is divided into slices and the slices are divided into segments; receiving compressed slices from the cloud storage system at the server; decompressing, at the server, the compressed slices; stitching the decompressed slices together to reconstruct the object; and returning the reconstructed object to the client.
10. The non-transitory storage medium of claim 9 , further comprising decompressing the compressed slices while streaming the compressed slices without landing the compressed slices at the server.
11. The non-transitory storage medium of claim 10 , further comprising reconstructing the decompressed slices at the server while streaming to the client.
12. A method for transferring objects to and from a cloud storage system, the method comprising:
receiving a request from a client at a server with regard to an object, wherein the client is configured to direct the request to an endpoint of the server;
determining a request type of the request by the server; when the request type is a GET request, performing the following data transfer operations, wherein the data transfer operations are configured to reduce an amount of data transferred between the server and the cloud storage system:
communicating, by the server, with the cloud storage system to request an object, wherein the object is divided into slices and the slices are divided into segments;
receiving compressed slices from the cloud storage system at the server;
decompressing, at the server, the compressed slices;
stitching the decompressed slices together to reconstruct the object; and
returning the reconstructed object to the client.
13. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform the method of claim 12 .