Data deduplication on a distributed file system using conditional writes

1. A method for deduplicating data on a distributed file system, the method comprising:

transmitting, by a client, a write request to a metadata server (“MDS”) associated with the distributed file system, wherein the MDS maintains location data for all data objects in the distributed file system, wherein the write request comprises an object identifier associated with a data object;

receiving, by the client, an object store location for an object store from the MDS;

transmitting, by the client, a metadata request to the object store using the object store location prior to transmitting the data object, wherein the metadata request includes the object identifier, and wherein the object store is separate from the MDS;

receiving, by the client, a metadata response from the object store to the metadata request;

determining, by the client, that the object store contains the data object when the metadata response includes an object designator and determining that the object store does not contain the data object when the metadata response does not include the object designator, where the data object is transmitted to the object store and stored at the object store when the metadata response does not include the object designator; and

transmitting, by the client, a commit request to the MDS, wherein the commit response includes the object designator, wherein the object identifier uniquely identifies the data object and the object designator allows the MDS to track how many instances of the data object are stored in the distributed file system.

2. The method of claim 1 , wherein the metadata request is a conditional write request or a conditional PUT request, wherein the data object is transmitted with the conditional write request or with the conditional PUT request.

3. The method of claim 1 , wherein the object designator uniquely identifies the data object.

4. The method of claim 1 , wherein the object store is a cloud object store.

5. A non-transitory computer readable storage medium comprising processor instructions for deduplicating data on a distributed file system, the instructions comprising:

transmitting, by a client, a write request to a metadata server (“MDS”) associated with the distributed file system, wherein the MDS maintains location data for all data objects in the distributed file system, wherein the write request comprises an object identifier associated with a data object;

receiving, by the client, an object store location for an object store from the MDS;

transmitting a metadata request to the object store using the object store location prior to transmitting the data object, wherein the metadata request includes the object identifier, and wherein the object store is separate from the MDS;

receiving, by the client, a metadata response from the object store to the metadata request;

determining, by the client, that the object store contains the data object when the metadata response includes an object designator and determining that the object store does not contain the data object when the metadata response does not include the object designator, where the data object is transmitted to the object store and stored at the object store when the metadata response does not include the object designator; and

transmitting, by the client, a commit request to the MDS, wherein the commit response includes the object designator, wherein the object identifier uniquely identifies the data object and the object designator allows the MDS to track how many instances of the data object are stored in the distributed file system.

6. The computer readable storage medium of claim 5 , wherein the conditional write request is a conditional PUT request.

7. The computer readable storage medium of claim 5 , wherein the object designator uniquely identifies the data object.

8. The computer readable storage medium of claim 5 , wherein the object store is a cloud object store.

9. A system for deduplicating data on a distributed file system, the system comprising a non-transitory computer readable medium and processor enabled to execute instructions for:

transmitting, by a client, a write request to a metadata server (“MDS”) associated with the distributed file system, wherein the MDS maintains location data for all data objects in the distributed file system, wherein the write request comprises an object identifier associated with a data object;

receiving, by the client, an object store location for an object store from the MDS;

transmitting a metadata request to the object store using the object store location prior to transmitting the data object, wherein the metadata request includes the object identifier, and wherein the object store is separate from the MDS;

receiving, by the client, a metadata response from the object store to the metadata request;

determining, by the client, that the object store contains the data object when the metadata response includes an object designator and determining that the object store does not contain the data object when the metadata response does not include the object designator, where the data object is transmitted to the object store and stored at the object store when the metadata response does not include the object designator; and

transmitting, by the client, a commit request to the MDS, wherein the commit response includes the object designator, wherein the object identifier uniquely identifies the data object and the object designator allows the MDS to track how many instances of the data object are stored in the distributed file system.

10. The system of claim 9 , wherein the conditional write request is a conditional PUT request.

11. The system of claim 9 , wherein the object designator uniquely identifies the data object.

12. The system of claim 9 , wherein the object store is a cloud object store.