File system consistency in a distributed system using version vectors

1. A method, comprising:

receiving, at a backup node of a distributed backup system comprising a plurality of backup nodes, a request to perform one or more file system operations on a file;

obtaining, by the backup node, a current version vector associated with the file from a first data repository and a last completed vector associated with the file from a second data repository, wherein the first data repository is configured to store current version vectors associated with files accessed by the plurality of backup nodes and the second data repository is configured to store last completed version vectors for each file accessed by the plurality of backup nodes, wherein a storage repository includes the first data repository and the second data repository, wherein the storage repository is separate from the plurality of backup nodes, wherein the current version vector associated with the file is comprised of a first incarnation identifier and a first transaction identifier and the last completed vector associated with the file is comprised of a second incarnation identifier and a second transaction identifier;

comparing, by the backup node, data of the current version vector with data of the last completed vector at least in part by:

comparing the first incarnation identifier of the current version vector associated with the file with the second incarnation identifier of the last completed version vector associated with the file, wherein an incarnation identifier indicates a number of times one of the one or more nodes of the distributed backup system has restarted; and

comparing the first transaction identifier of the current version vector associated with the file with the second transaction identifier of the last completed version vector associated with the file, wherein a transaction identifier indicates a number of transactions associated with the incarnation identifier;

determining, by the backup node, that the data of the current version vector is greater than the data of the last completed version vector, wherein the data of the current version vector being greater than the data of the last completed version vector indicates the file is in an inconsistent state;

in response to determining that the current version vector comprising the first incarnation identifier and the first transaction identifier is the greater than the data of the last completed version vector comprising the second incarnation identifier and the second transaction identifier, waiting to access the file until the current version vector is less than or equal to the last completed version vector stored in the second data repository; and

in response to the current version vector comprising the first incarnation identifier and the first transaction identifier being less than or equal to the data of the last completed version vector comprising the second incarnation identifier and the second transaction identifier accessing the file at least in part by:

reading the file;

determining that an inconsistency for the file exists; and

fixing the inconsistency at least in part by rolling back the file to a previous stable version of the file.

2. The method of claim 1 , wherein fixing the inconsistency in the file comprises attempting to update the file based on a cached copy of a file update.

3. The method of claim 1 , further comprising performing the one or more file system operations on the file, wherein performing the one or more file system operations on the file comprises:

generating a vector clock for the backup node, the vector clock including a corresponding incarnation identifier and a corresponding incremented transaction identifier relative to a previous transaction performed by the backup node;

updating the current version vector associated with the file based on the generated vector clock;

performing one or more read or write operations on the file; and

upon completion of the one or more read or write operations, updating the latest completed version vector associated with the file based on the generated vector clock.

4. The method of claim 1 , further comprising:

restarting the backup node;

obtaining an incarnation identifier from a previous version vector associated with the backup node; and

resetting the first incarnation identifier and the first transaction identifier, wherein resetting includes incrementing the first incarnation identifier relative to a previous incarnation identifier and setting the first transaction identifier to an initial, sequential value.

5. The method of claim 1 , wherein the current version vector and last completed version vector further comprises a unique identifier of the backup node.

6. A computer program product, the computer program product being embodied in a non-transitory computer readable medium and comprising instructions for:

receiving, at a backup node of a distributed backup system comprising a plurality of backup nodes, a request to perform one or more file system operations on a file;

obtaining, by the backup node, a current version vector associated with the file from a first data repository and a last completed vector associated with the file from a second data repository, wherein the first data repository is configured to store current version vectors associated with files accessed by the plurality of backup nodes and the second data repository is configured to store last completed version vectors for each file accessed by the plurality of backup nodes, wherein a storage repository includes the first data repository and the second data repository, wherein the storage repository is separate from the plurality of backup nodes, wherein the current version vector associated with the file is comprised of a first incarnation identifier and a first transaction identifier and the last completed vector associated with the file is comprised of a second incarnation identifier and a second transaction identifier;

comparing, by the backup node, data of the current version vector with data of the last completed vector at least in part by:

comparing the first incarnation identifier of the current version vector associated with the file with the second incarnation identifier of the last completed version vector associated with the file, wherein an incarnation identifier indicates a number of times one of the one or more nodes of the distributed backup system has restarted; and

comparing the first transaction identifier of the current version vector associated with the file with the second transaction identifier of the last completed version vector associated with the file, wherein a transaction identifier indicates a number of transactions associated with the incarnation identifier;

determining, by the backup node, that the data of the current version vector is greater than the data of the last completed version vector, wherein the data of the current version vector being greater than the data of the last completed version vector indicates the file is in an inconsistent state;

in response to determining that the current version vector comprising the first incarnation identifier and the first transaction identifier is the greater than the data of the last completed version vector comprising the second incarnation identifier and the second transaction identifier, waiting to access the file until the current version vector is less than or equal to the last completed version vector stored in the second data repository; and

in response to the current version vector comprising the first incarnation identifier and the first transaction identifier being less than or equal to the data of the last completed version vector comprising the second incarnation identifier and the second transaction identifier, accessing the file at least in part by:

reading the file;

determining that an inconsistency for the file exists; and

fixing the inconsistency at least in part by rolling back the file to a previous stable version of the file.

7. The computer program product of claim 6 , wherein fixing the inconsistency in the file comprises attempting to update the file based on a cached copy of a file update.

8. The computer program product of claim 6 , further comprising performing the one or more file system operations on the file, wherein performing the one or more file system operations on the file comprises:

generating a vector clock for the backup node, the vector clock including a corresponding incarnation identifier and a corresponding incremented transaction identifier relative to a previous transaction performed by the backup node;

updating the current version vector associated with the file based on the generated vector clock;

performing one or more read or write operations on the file; and

upon completion of the one or more read or write operations, updating the latest completed version vector associated with the file based on the generated vector clock.

9. The computer program product of claim 6 , further comprising instructions for:

restarting the backup node;

obtaining an incarnation identifier from a previous version vector associated with the node; and

resetting the first incarnation identifier and the first transaction identifier, wherein resetting includes incrementing the first incarnation identifier relative to a previous incarnation identifier and setting the first transaction identifier to an initial, sequential value.

10. The computer program product of claim 6 , wherein the current version vector and last completed version vector further comprises a unique identifier of the backup node.

11. A system comprising:

a processor configured to:

receive a request to perform one or more file system operations on a file;

obtain a current version vector associated with the file from a first data repository and a last completed vector associated with the file from a second data repository, wherein the first data repository is configured to store current version vectors associated with files accessed by a plurality of backup nodes and the second data repository is configured to store last completed version vectors for each file accessed by the plurality of backup nodes, wherein a storage repository includes the first data repository and the second data repository, wherein the storage repository is separate from the plurality of backup nodes, wherein the current version vector associated with the file is comprised of a first incarnation identifier and a first transaction identifier and the last completed vector associated with the file is comprised of a second incarnation identifier and a second transaction identifier;

compare data of the current version vector with data of the last completed vector, wherein to compare data of the current version vector with data of the last completed vector, the processor is configured to:

compare the first incarnation identifier of the current version vector associated with the file with the second incarnation identifier of the last completed version vector associated with the file, wherein an incarnation identifier indicates a number of times one of the one or more nodes of a distributed backup system has restarted; and

compare the first transaction identifier of the current version vector associated with the file with the second transaction identifier of the last completed version vector associated with the file, wherein a transaction identifier indicates a number of transactions associated with the incarnation identifier;

determine that the data of the current version vector is greater than the data of the last completed version vector, wherein the data of the current version vector being greater than the data of the last completed version vector indicates the file is in an inconsistent state;

in response to a determination that the current version vector comprising the first incarnation identifier and the first transaction identifier is the greater than the data of the last completed version vector comprising the second incarnation identifier and the second transaction identifier, wait to access the file until the current version vector is less than or equal to the last completed version vector stored in the second data repository; and

in response to the current version vector comprising the first incarnation identifier and the first transaction identifier being less than or equal to the data of the last completed version vector comprising the second incarnation identifier and the second transaction identifier, access the file at least in part by:

read the file;

determine that an inconsistency for the file exists; and

fix the inconsistency at least in part by rolling back the file to a previous stable version of the file; and

a memory coupled to the processor and configured to provide the processor with instructions.

12. The system of claim 11 , wherein to fix the inconsistency, the processor is configured to attempt to update the file based on a cached copy of a file update.

13. The system of claim 11 , wherein the processor is further configured to perform the one or more file system operations on the file, wherein to perform the one or more file system operations on the file, the processor is further configured to:

generate a vector clock for a backup node, the vector clock including a corresponding incarnation identifier and a corresponding incremented transaction identifier relative to a previous transaction performed by the backup node;

update the current version vector associated with the file based on the generated vector clock;

perform one or more read or write operations on the file; and

upon completion of the one or more read or write operations, update the latest completed version vector associated with the file based on the generated vector clock.

14. The system of claim 11 , wherein the processor is further configured to:

restart a backup node;

obtain an incarnation identifier from a previous version vector associated with the backup node; and

reset the first incarnation identifier and the first transaction identifier, wherein to reset the first incarnation identifier and the first transaction identifier, the processor is further configured to increment the first incarnation identifier relative to a previous incarnation identifier and setting the first transaction identifier to an initial, sequential value.

15. The system of claim 11 , wherein the current version vector and last completed version vector further comprises a unique identifier of a backup node.