Disaster recovery in a distributed file system

1. A method for repairing a metadata server (“MDS”) in a distributed file system, comprising:

taking a snapshot of the MDS, wherein the MDS maintains metadata including authoritative object store locations for a plurality of data objects, wherein the MDS comprises a map identifying the object store location for each of the plurality of data objects and a queue identifying individuals of the plurality of data objects that only reside on a transient object store, wherein clients access the MDS to identify locations of the data objects stored in the authoritative object store locations and the transient object store;

flushing at least first data objects selected from a plurality of data objects stored in a transient object store to an authoritative object store after taking the snapshot;

after determining that the MDS requires repair, receiving the snapshot at the MDS from a backup storage location at a point-in-time subsequent to taking the snapshot and subsequent to flushing the first data objects in the transient object store,

wherein the snapshot includes a snapshot map and a snapshot queue corresponding to the map and the queue at the time of the snapshot, wherein the MDS is separate from the transient object store and separate from authoritative object stores that store the plurality of data objects;

restoring the map and the snapshot queue of the MDS based on the snapshot, wherein the snapshot queue indicates that the first data objects flushed from the transient object store to the authoritative object store are stored in the transient object store;

correcting the locations in the MDS of the first data objects identified in the snapshot queue that are incorrect by:

locating the first data objects identified in the snapshot queue on the distributed file system; and

updating the map of the restored MDS to identify the object store locations for the first data objects identified in the snapshot queue.

2. The method of claim 1 , further comprising transferring the individuals of the plurality of data objects identified in the queue from the transient object store to an authoritative object store prior to the point-in-time.

3. The method of claim 1 , wherein the map comprises a bitmap having a bit for every object store location in the distributed file system.

4. The method of claim 1 , wherein the plurality of data objects are associated with sequential data object identifiers.

5. The method of claim 1 , further comprising taking the snapshot of a MDS.

6. The method of claim 5 , further comprising transmitting the snapshot to a backup storage location.

7. A non-transitory computer readable medium comprising processor instructions for repairing a metadata server (“MDS”) in a distributed file system, the instructions comprising:

taking a snapshot of the MDS, wherein the MDS maintains metadata including authoritative object store locations for a plurality of data objects, wherein the MDS comprises a map identifying the object store location for each of the plurality of data objects and a queue identifying individuals of the plurality of data objects that only reside on a transient object store, wherein clients access the MDS to identify locations of the data objects stored in the authoritative object store locations and the transient object store;

flushing at least first data objects selected from a plurality of data objects stored in a transient object store to an authoritative object store after taking the snapshot;

after determining that the MDS requires repair, receiving the snapshot at the MDS from a backup storage location at a point-in-time subsequent to taking the snapshot and subsequent to flushing the first data objects in the transient object store,

wherein the snapshot includes a snapshot map and a snapshot queue corresponding to the map and the queue at the time of the snapshot, wherein the MDS is separate from the transient object store and separate from authoritative object stores that store the plurality of data objects;

restoring the map and the snapshot queue of the MDS based on the snapshot, wherein the snapshot queue indicates that the first data objects flushed from the transient object store to the authoritative object store are stored in the transient object store;

correcting the locations in the MDS of the first data objects identified in the snapshot queue that are incorrect by:

locating the first data objects identified in the snapshot queue on the distributed file system; and

updating the map of the restored MDS to identify the object store locations for the first data objects identified in the snapshot queue.

8. The non-transitory computer readable medium of claim 7 , further comprising instructions for transferring the individuals of the plurality of data objects identified in the queue from the transient object store to an authoritative object store prior to the point-in-time.

9. The non-transitory computer readable medium of claim 7 , wherein the map comprises a bitmap having a bit for every object store location in the distributed file system.

10. The non-transitory computer readable medium of claim 7 , wherein the plurality of data objects are associated with sequential data object identifiers.

11. A system for repairing a metadata server (“MDS”) in a distributed file system, the system comprising a non-transitory computer readable medium and processor enabled to execute instructions for:

taking a snapshot of the MDS, wherein the MDS maintains metadata including authoritative object store locations for a plurality of data objects, wherein the MDS comprises a map identifying the object store location for each of the plurality of data objects and a queue identifying individuals of the plurality of data objects that only reside on a transient object store, wherein clients access the MDS to identify locations of the data objects stored in the authoritative object store locations and the transient object store;

flushing at least first data objects selected from a plurality of data objects stored in a transient object store to an authoritative object store after taking the snapshot;

after determining that the MDS requires repair, receiving the snapshot at the MDS from a backup storage location at a point-in-time subsequent to taking the snapshot and subsequent to flushing the first data objects in the transient object store,

wherein the snapshot includes a snapshot map and a snapshot queue corresponding to the map and the queue at the time of the snapshot, wherein the MDS is separate from the transient object store and separate from authoritative object stores that store the plurality of data objects;

restoring the map and the snapshot queue of the MDS based on the snapshot, wherein the snapshot queue indicates that the first data objects flushed from the transient object store to the authoritative object store are stored in the transient object store;

correcting the locations in the MDS of the first data objects identified in the snapshot queue that are incorrect by:

locating the first data objects identified in the snapshot queue on the distributed file system; and

updating the map of the restored MDS to identify the object store locations for the first data objects identified in the snapshot queue.

12. The system of claim 11 , further comprising transferring the individuals of the plurality of data objects identified in the queue from the transient object store to an authoritative object store prior to the point-in-time.

13. The system of claim 11 , wherein the map comprises a bitmap having a bit for every object store location in the distributed file system.

14. The system of claim 11 , wherein the plurality of data objects are associated with sequential data object identifiers.

15. The system of claim 11 , further comprising taking the snapshot of a MDS.

16. The system of claim 15 , further comprising transmitting the snapshot to a backup storage location.