De-duplicating distributed file system using cloud-based object store
Techniques to provide a de-duplicating distributed file system using a cloud-based object store are disclosed. In various embodiments, a request to store a file comprising a plurality of chunks of file data is received. A determination to store at least a subset of the plurality of chunks is made. The request is responded to at least in part by providing an indication to store two or more chunks comprising the at least a subset of the plurality of chunks comprising the file as a single stored object that includes the combined chunk data of said two or more chunks.
1. A method of storing file system data, comprising:
receiving, at a processor, a request to store a file comprising a plurality of chunks of file data;
determining, by the processor, to store at least a first subset of the plurality of chunks of file data based at least in part on a chunk identifier, wherein a second subset of the plurality of chunks of file data are already stored at a remote storage;
determining, by the processor, a deduplication chunk size for the plurality of chunks of file data of the first subset, wherein the deduplication chunk size facilitates achieving a desired deduplication performance when storing the first subset of the plurality of chunks of file data, and wherein the deduplication chunk size is larger than a chunk size of a chunk included in the first subset of the plurality of chunks;
selecting, by the processor, which chunks of the first subset of the plurality of chunks of file data to combine into a single stored object that satisfies the deduplication chunk size associated with the desired deduplication performance;
combining, by the processor, the selected chunks of the first subset of the plurality of chunks of file data into the single stored object satisfying the deduplication chunk size; and
providing, by the processor, the single stored object that includes the combined selected chunks of the first subset of the plurality of chunks of file data to the remote storage, wherein the remote storage is configured to store the provided single stored object, and wherein storing the single stored object achieves a better deduplication performance than would be achieved if the combined selected chunks were stored individually.
2. The method of claim 1 , wherein the request is received from a file system client.
3. The method of claim 1 , wherein the request includes a hash or other representation of the chunks comprising the file.
4. The method of claim 3 , wherein the hash or other representation is used to determine whether a chunk comprising the file has already been stored by the file system.
5. The method of claim 4 , wherein determining to store the at least a first subset of the plurality of chunks includes using the respective hash or other representation of the chunks comprising the at least a first subset of the plurality of chunks to determine that said chunks have not yet been stored.
6. The method of claim 1 , further comprising responding to the request at least in part by providing an indication to store two or more chunks comprising the at least a first subset of the plurality of chunks comprising the file as the single stored object that includes combined chunk data of said two or more chunks, wherein said indication to store two or more chunks comprising the at least a subset of the plurality of chunks comprising the file as the single stored object includes a locator or other identifier associated with the single stored object and data associating said at least a subset of the plurality of chunks comprising the file with the locator or other identifier.
7. The method of claim 6 , wherein the locator or other identifier indicates a location to which the single stored object should be sent to be stored.
8. The method of claim 1 , wherein the request is sent by a file system client that is configured to respond to said response to the request by assembling the single stored object using two or more of the first subset of the plurality of chunks.
9. The method of claim 1 , further comprising storing metadata that associates said combined selected chunks with said single stored object.
10. The method of claim 9 , wherein said metadata includes for each chunk a location of data comprising that chunk with the single stored object.
11. The method of claim 1 , further comprising verifying storage of said combined selected chunks at least in part by retrieving said single stored object and using chunk data of the respective chunks comprising the single stored object to verify the chunk data as stored corresponds to the chunk data that was expected to be stored.
12. The method of claim 1 , further comprising storing for the single stored object a reference count that reflects a number of chunks comprising a blob that remain subject to being retained by a file system.
13. The method of claim 12 , further comprising decrementing the reference count based at least in part on a determination that a chunk comprising the blob no longer remains subject to being retained by a file system.
14. A system to store file system data, comprising:
a communication interface; and
a processor coupled to the communication interface and configured to:
receive via the communication interface a request to store a file comprising a plurality of chunks of file data;
determine to store at least a first subset of the plurality of chunks of file data based at least in part on a chunk identifier, wherein a second subset of the plurality of chunks of file data are already stored at a remote storage;
determine a deduplication chunk size for the plurality of chunks of file data of the first subset, wherein the deduplication chunk size facilitates achieving a desired deduplication performance when storing the first subset of the plurality of chunks of file data, and wherein the deduplication chunk size is larger than a chunk size of a chunk included in the first subset of the plurality of chunks;
select which chunks of the first subset of the plurality of chunks of file data to combine into a single stored object that satisfies the deduplication chunk size associated with the desired deduplication performance;
combine the selected chunks of the first subset of the plurality of chunks of file data into the single stored object satisfying the deduplication chunk size; and
provide the single stored object that includes the combined selected chunks of the first subset of the plurality of chunks of file data to the remote storage, wherein the remote storage is configured to store the provided single stored object, and wherein storing the single stored object achieves a better deduplication performance than would be achieved if the combined selected chunks were stored individually.
15. The system of claim 14 , wherein the request includes a hash or other representation of the chunks comprising the file.
16. The system of claim 15 , wherein the hash or other representation is used to determine whether a chunk comprising the file has already been stored.
17. The system of claim 16 , wherein determining to store the at least a subset of the plurality of chunks includes using the respective hash or other representation of the chunks comprising the at least a subset of the plurality of chunks to determine that said chunks have not yet been stored.
18. The system of claim 14 , wherein the processor is further configured to verify storage of said combined selected chunks at least in part by retrieving said single stored object and using chunk data of the respective chunks comprising the single stored object to verify the chunk data as stored corresponds to the chunk data that was expected to be stored.
19. A computer program product to store file system data, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving a request to store a file comprising a plurality of chunks of file data;
determining to store at least a first subset of the plurality of chunks of file data based at least in part on a chunk identifier, wherein a second subset of the plurality of chunks of file data are already stored at a remote storage;
determining a deduplication chunk size for the plurality of chunks of file data of the first subset, wherein the deduplication chunk size facilitates achieving a desired deduplication performance when storing the first subset of the plurality of chunks of file data, and wherein the deduplication chunk size is larger than a chunk size of a chunk included in the first subset of the plurality of chunks;
selecting which chunks of the first subset of the plurality of chunks of file data to combine into a single stored object that satisfies the deduplication chunk size associated with the desired deduplication performance;
combining the selected chunks of the first subset of the plurality of chunks of file data into the single stored object satisfying the deduplication chunk size; and
providing the single stored object that includes the combined selected chunks of the first subset of the plurality of chunks of file data to the remote storage, wherein the remote storage is configured to store the provided single stored object, and wherein storing the single stored object achieves a better deduplication performance than would be achieved if the combined selected chunks were stored individually.
20. The method of claim 1 , wherein the deduplication chunk size is determined based at least in part on an overhead associated with computing and storing hashes associated with the plurality of chunks.