Efficient change block tracking in a distributed generation-id (GID) based log structured file system (LSFS)

1. A method comprising:

performing a plurality of iterations of write operations on portions of a file tree of a file system to a first storage cluster, each write operation flagging the written portion of the file tree with a generation identifier indicative of an associated iteration of the write operations;

identifying a set of the written portions of the file tree as unsynchronized between the first storage cluster and a second storage cluster by performing a search for changes made to the written portions of the file tree between a generation identifier associated with the first storage cluster and a generation identifier associated with the second storage cluster; and

synchronizing the first and second storage clusters by sending, via a network, the unsynchronized portions of the file tree from the first storage cluster to the second storage cluster.

2. The method as in claim 1 , wherein identifying the set of unsynchronized portions of the file tree between the first storage cluster and the second storage cluster comprises:

searching for changes made to the written portions of the file tree for a particular inode.

3. The method as in claim 1 , wherein the unsynchronized portions of the file tree are objects or blocks.

4. The method as in claim 1 , further comprising:

copying portions of the file tree associated with a particular generation identifier from the first storage cluster;

identifying a set of changed portions of the file tree on the second storage cluster between the particular generation identifier and a current generation identifier used by the second storage cluster; and

restoring the copied portions of the file tree on the first storage cluster by transmitting, via the network, the identified set of changed portions of the file tree between the particular generation identifier and the current generation identifier from the second storage cluster to the first storage cluster.

5. The method as in claim 1 , further comprising:

performing a failback from the second storage cluster to the first storage cluster by using the generation identifiers to identify a set of the portions of the file tree written to the second storage cluster after a failover from the first storage cluster to the second storage cluster.

6. The method as in claim 1 , further comprising:

performing a local backup on the first storage cluster of a set of the portions of the file tree written to the first storage cluster and flagged with a particular generation identifier;

making changes to the locally backed up set on the first storage cluster in part by flagging the changes with a new generation identifier; and

propagating the changes to the locally backed up set to the second storage cluster, based in part on a comparison between the particular generation identifier and the new generation identifier.

7. The method as in claim 1 , wherein the file system is a log structured file system (LSFS).

8. An apparatus, comprising:

one or more network interfaces to communicate with a network;

a processor coupled to the network interfaces and configured to execute one or more processes; and

a memory configured to store a process executable by the processor, the process when executed configured to:

perform a plurality of iterations of write operations on portions of a file tree of a file system to a first storage cluster, each write operation flagging the written portion of the file tree with a generation identifier indicative of an associated iteration of the write operations;

identify a set of the written portions of the file tree as unsynchronized between the first storage cluster and a second storage cluster by performing a search for changes made to the written portions of the file tree between a generation identifier associated with the first storage cluster and a generation identifier associated with the second storage cluster; and

synchronize the first and second storage clusters by sending, via a network, the unsynchronized portions of the file tree from the first storage cluster to the second storage cluster.

9. The apparatus as in claim 8 , wherein the apparatus identifies the set of unsynchronized portions of the file tree between the first storage cluster and the second storage cluster by:

searching for changes made to the written portions of the file tree for a particular inode.

10. The apparatus as in claim 8 , wherein the unsynchronized portions of the file tree are objects or blocks.

11. The apparatus as in claim 8 , wherein the process when executed is further configured to:

copying portions of the file tree associated with a particular generation identifier from the first storage cluster;

identify a set of changed portions of the file tree on the second storage cluster between the particular generation identifier and a current generation identifier used by the second storage cluster; and

restore the copied portions of the file tree on the first storage cluster by transmitting, via the network, the identified set of changed portions of the file tree between the particular generation identifier and the current generation identifier from the second storage cluster to the first storage cluster.

12. The apparatus as in claim 8 , wherein the process when executed is further configured to:

perform a failback from the second storage cluster to the first storage cluster by using the generation identifiers to identify a set of the portions of the file tree written to the second storage cluster after a failover from the first storage cluster to the second storage cluster.

13. The apparatus as in claim 8 , wherein the process when executed is further configured to:

perform a local backup on the first storage cluster of a set of the portions of the file tree written to the first storage cluster and flagged with a particular generation identifier;

make changes to the locally backed up set on the first storage cluster in part by flagging the changes with a new generation identifier; and

propagate the changes to the locally backed up set to the second storage cluster, based in part on a comparison between the particular generation identifier and the new generation identifier.

14. The apparatus as in claim 8 , wherein the file system is a log structured file system (LSFS).

15. The apparatus as in claim 8 , wherein the network comprises a storage area network (SAN), local area network (LAN), or PCI-Express (PCIe) network.

16. A tangible, non-transitory, computer-readable medium storing program instructions that cause a device to execute a process comprising:

performing a plurality of iterations of write operations on portions of a file tree of a file system to a first storage cluster, each write operation flagging the written portion of the file tree with a generation identifier indicative of an associated iteration of the write operations;

identifying a set of the written portions of the file tree as unsynchronized between the first storage cluster and a second storage cluster by performing a search for changes made to the written portions of the file tree between a generation identifier associated with the first storage cluster and a generation identifier associated with the second storage cluster; and

synchronizing the first and second storage clusters by sending, via a network, the unsynchronized portions of the file tree from the first storage cluster to the second storage cluster.

17. The computer-readable medium as in claim 16 , wherein identifying the set of unsynchronized portions of the file tree between the first storage cluster and the second storage cluster comprises:

searching for changes made to the written portions of the file tree for a particular inode.

18. The computer readable medium as in claim 16 , wherein the process further comprises:

copying portions of the file tree associated with a particular generation identifier from the first storage cluster;

identifying a set of changed portions of the file tree on the second storage cluster between the particular generation identifier and a current generation identifier used by the second storage cluster; and

restoring the copied portions of the file tree on the first storage cluster by transmitting, via the network, the identified set of changed portions of the file tree between the particular generation identifier and the current generation identifier from the second storage cluster to the first storage cluster.

19. The computer readable medium as in claim 16 , wherein the process further comprises:

performing a failback from the second storage cluster to the first storage cluster by using the generation identifiers to identify a set of the portions of the file tree written to the second storage cluster after a failover from the first storage cluster to the second storage cluster.

20. The computer readable medium as in claim 16 , wherein the process further comprises:

performing a local backup on the first storage cluster of a set of the portions of the file tree written to the first storage cluster and flagged with a particular generation identifier;

making changes to the locally backed up set on the first storage cluster in part by flagging the changes with a new generation identifier; and

propagating the changes to the locally backed up set to the second storage cluster, based in part on a comparison between the particular generation identifier and the new generation identifier.