Data deduplication on a distributed file system using conditional writes
A write request is transmitted from a client to a metadata server (“MDS”), wherein the write request comprises an object identifier associated with a data object. An object store location for an object store is received from the MDS. A conditional write request is transmitted to the object store using the object store location, wherein the conditional write request includes the object identifier and the data object. A metadata response is received from the object store. The metadata response is determined to contain an object designator. A commit request is transmitted to the MDS, wherein the commit response includes the object designator.
1. A method for deduplicating data on a distributed file system, the method comprising:
writing a data object to an object store by a first client, wherein the data object is associated with a first object designator, the first object designator includes a count, and the object store implements object versioning;
writing the data object to the object store by a second client, wherein the data object is associated with a second object designator and the second object designator includes with a count;
receiving a delete instruction from the first client; and
deleting a version of the data object associated with the first client without deleting a version associated with the second client such that the data object is available in the distributed file system.
2. The method of claim 1 , further comprising deleting a latest version of the data object.
3. The method of claim 1 , further comprising receiving a metadata request from the first client and a metadata request from the second client.
4. A method for deduplicating data on a distributed file system, the method comprising:
writing a data object to an object store by a first client;
updating an in-flight data structure at a metadata server to maintain a reference to the data object until the data object is committed to the object store;
receiving a write request from a second client to write the data object to the object store;
informing the second client that the data object is in-flight;
returning an object designator of the data object to the second client, wherein the object designator includes a count;
committing the data object to the object store; and
updating the metadata server.
5. The method of claim 1 , wherein the writing of the data object to an object store by a first client and the writing the data object to the object store by a second client occur at substantially the same time.
6. The method of claim 5 , further comprising enabling object versioning at an object store.
7. The method of claim 1 , wherein only one version of a data object can be deleted at a time.
8. The method of claim 1 , further comprising receiving instructions from a metadata server to delete the data object.
9. The method of claim 8 , wherein the metadata server instructs the deletion of the data object when performing a garbage collection and upon an object designator count equaling zero, wherein the object designator is a count associated with the number of instances an object designator has been added by the client.
10. The method of claim 1 , further comprising deleting a final version of the data object.
11. The method of claim 9 , further comprising deleting the data object after all versions of the data object have been deleted such that the data object is no longer available.
12. The method of claim 4 , wherein the writing of a data object to an object store by a first client and receiving a write request from a second client occur at substantially the same time.
13. The method of claim 4 , further comprising updating the reference to an in-flight data structure to reference a standard mapping once the metadata server has been updated.
14. The method of claim 4 , wherein the method refrains from issuing two object designators.
15. The method of claim 4 , further comprising determining that the data object already exists in the data store and refraining from writing a second instance of the data object to the object store.
16. The method of claim 4 , wherein the first write request or the second write request is a conditional write request or a conditional PUT request, wherein the data object is transmitted with the conditional write request or with the conditional PUT request.
17. The method of claim 4 , further comprising informing the second client that the data object is in-flight and returning the object designator.
18. The method of claim 4 , wherein the in-flight data structure maps an object identifier to the object designator.
19. The method of claim 4 , further comprising incrementing an object designator count associated with the data object for both the first and second request to write the data object from the first and second clients.
20. The method of claim 19 , further comprising decrementing the object designator count when a client requests a data object to be deleted wherein the data object will still be available until the counter is decremented to zero.