System and method of generating automatic checkpoints of a distributed file system
Disclosed herein are techniques for generating fractional checkpoints on a distributed file system by snapshotting subcomponents referred to as “file sets.” The techniques include capturing a present time; calculating from configured parameters a future wakeup time for a subsequent set of checkpoints from the present time; walking a database of meta file set objects to locate at least one meta file set object; calculating a retention period for a snapshot that is being created; and generating a global data-less snapshot for the meta file set object and remote data file set objects associated with the meta file set object, and then repeating the process for subsequent file set objects in the database.
1. A computer implemented method of generating a global checkpoint, the method comprising:
capturing a present time;
calculating from configured parameters a future wakeup time for a subsequent set of snapshots from the present time;
walking a database of meta file set objects to locate at least one meta file set object of a plurality of meta file set objects;
calculating a retention period for a snapshot that is being created; and
generating a global data-less snapshot for the at least one meta file set object and remote data file set objects associated with the at least one meta file set object,
wherein the at least one meta file set object describes a data structure that houses a name space of a spanning file set object describing a distinct sub-filesystem inside a distributed filesystem, and
wherein the remote data file set objects include a data structure that houses data associated with a file of the at least one meta file set object and are associated with the at least one meta file set object spanning two or more nodes.
2. The computer implemented method as in claim 1 , further comprising:
performing an incremental distributed recovery on only a subset of the two or more nodes spanned by the remote data file set objects.
3. The computer implemented method as in claim 1 , further comprising:
saving node location information for the at least one meta file set object where the global data-less snapshot originated.
4. The computer implemented method as in claim 1 , further comprising:
saving node location information for each data file set object that has also participated in the global data-less snapshot.
5. The computer implemented method as in claim 1 , further comprising:
saving a status of whether a file set being snapshotted is a meta file set object or a data file set object.
6. The computer implemented method as in claim 1 , further comprising:
saving a master file checksum for the meta file set object and data file set objects.
7. The computer implemented method as in claim 1 , further comprising:
editing a root namespace objects list of data file set objects to install a previous snapshot.
8. A system for generating a global checkpoint comprising:
a computer processing device; and
a global snapshot module stored within the computer processing device which performs the following operations:
capturing a present time;
calculating from configured parameters a future wakeup time for a subsequent set of snapshots from the present time;
walking a database of meta file set objects to locate at least one of a plurality of meta file set objects;
calculating a retention period for a snapshot that is being created; and
generating a global data-less snapshot for at least one meta file set object and remote data file set objects associated with the at least one meta file set object,
wherein the at least one meta file set object describes a data structure that houses a name space of a spanning file set object describing a distinct sub-filesystem inside a distributed filesystem, and
wherein the remote data file set objects include a data structure that houses data associated with a file of the at least one meta file set object and are associated with the at least one meta file set object spanning two or more nodes.
9. The system as in claim 8 , further comprising:
a filesystem recovery module configured to perform an incremental distributed recovery on only a subset of the two or more nodes spanned by the remote data file set objects.
10. The system as in claim 9 , wherein the filesystem recovery module is further configured to save node location information for the at least one meta file set object where the global data-less snapshot originated.
11. The system as in claim 9 , wherein the filesystem recovery module is further configured to save node location information for each data file set object that has also participated in the global data-less snapshot.
12. The system as in claim 9 , wherein the filesystem recovery module is further configured to save a status of whether a file set being snapshotted is a meta file set object or a data file set object.
13. The system as in claim 9 , wherein the filesystem recovery module is further configured to save a master file checksum for the meta file set object and data file set objects.
14. The system as in claim 9 , wherein the filesystem recovery module is further configured to edit a root namespace objects list of data file set objects to install a previous snapshot.
15. A non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform a global checkpoint operation, the operation comprising:
capturing a present time;
calculating from configured parameters a future wakeup time for a subsequent set of snapshots from the present time;
walking a database of meta file set objects to locate at least one of a plurality of meta file set objects;
calculating a retention period for a snapshot that is being created; and
generating a global data-less snapshot for at least one meta file set object and remote data file set objects associated with the at least one meta file set object,
wherein the at least one meta file set object describes a data structure that houses a name space of a spanning file set object describing a distinct sub-filesystem inside a distributed filesystem, and
wherein the remote data file set objects include a data structure that houses data associated with a file of the at least one meta file set object and are associated with the at least one meta file set object spanning two or more nodes.
16. The non-transitory computer-readable medium as in claim 15 , wherein the instructions further cause the one or more processors to:
perform an incremental distributed recovery on only a subset of the two or more nodes spanned by the remote data file set objects.
17. The non-transitory computer-readable medium as in claim 15 , wherein the instructions further cause the one or more processors to:
edit a root namespace objects list of data file set objects to install a previous snapshot.