Efficient transfer to and from a deduplicated cloud storage system
Efficient data transfer is disclosed. A server provides an endpoint for a client. The client can communicate with a cloud storage system through the server rather than connect directly to the cloud storage system. The server is configured to perform requests in a manner that reduces the amount of data transferred to and from the cloud storage system.
1. A method for transferring data to a cloud storage system from a client, the method comprising:
receiving a request from a client at a server, wherein the request is associated with an object to be placed in the cloud storage system;
processing the object, by the server, to generate segments and to generate fingerprints that include a fingerprint for each of the segments;
sending the fingerprints, by the server, to the cloud storage system, wherein the cloud storage system compares the fingerprints received from the server with fingerprints stored at the cloud storage system;
receiving a list of missing fingerprints at the server from the cloud storage system, wherein the list of missing fingerprints identifies segments of the object that are not stored at the cloud storage system; and
sending, by the server, the segments corresponding to the list of missing fingerprints to the cloud storage system.
2. The method of claim 1 , further comprising processing the object by dividing the object into one or more slices.
3. The method of claim 2 , further comprising processing the object by dividing each of the one or more slices into segments.
4. The method of claim 1 , further comprising compressing the segments corresponding to the list of missing fingerprints.
5. The method of claim 1 , further comprising incorporating the segments corresponding to the list of missing fingerprints into the cloud storage system.
6. The method of claim 1 , further comprising communicating with the cloud storage system, by the server, using remote procedure calls.
7. The method of claim 1 , further comprising configuring the client to point to an endpoint of the server.
8. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
receiving a request from a client at a server, wherein the request is associated with an object to be placed in the cloud storage system;
processing the object, by the server, to generate segments and to generate fingerprints that include a fingerprint for each of the segments;
sending the fingerprints, by the server, to the cloud storage system, wherein the cloud storage system compares the fingerprints received from the server with fingerprints stored at the cloud storage system;
receiving a list of missing fingerprints at the server from the cloud storage system, wherein the list of missing fingerprints identifies segments of the object that are not stored at the cloud storage system; and
sending, by the server, the segments corresponding to the list of missing fingerprints to the cloud storage system.
9. The non-transitory storage medium of claim 8 , the operations further comprising processing the object by dividing the object into one or more slices.
10. The non-transitory storage medium of claim 9 , the operations further comprising processing the object by dividing each of the one or more slices into segments.
11. The non-transitory storage medium of claim 8 , the operations further comprising compressing the segments corresponding to the list of missing fingerprints.
12. The non-transitory storage medium of claim 8 , the operations further comprising incorporating the segments corresponding to the list of missing fingerprints into the cloud storage system.
13. The non-transitory storage medium of claim 8 , the operations further comprising communicating with the cloud storage system, by the server, using remote procedure calls.
14. The non-transitory storage medium of claim 8 , the operations further comprising configuring the client to point to an endpoint of the server.