Methods and systems for file replication utilizing differences between versions of files
Methods and systems for efficient file replication are provided. In some embodiments, one or more coarse signatures for blocks in a base file are compared with those coarse signatures for blocks of a revised file, until a match is found. A fine signature is then generated for the matching block of the revised file and compared to a fine signature of the base file. Thus, fine signatures are not computed unless a coarse signature match has been found, thereby minimizing unneeded time-consuming fine signature calculations. Methods are also provided for determining whether to initiate a delta file generation algorithm, or whether to utilize a more efficient replication method, based upon system and/or file parameters. In accordance with additional embodiments, the lengths of valid data on physical blocks are obtained from physical block mappings for the files, and these lengths and mappings are utilized for delta file generation, to minimize unnecessary signature computations.
1. A method for determining whether to create a delta file reflecting differences between a base file and a revised version of the base file, the method comprising:
obtaining, by a computer system, one or more system parameters relating to resources of the computer system;
determining, by the computer system, a threshold change amount between the base file and the revised version of the base file, wherein the threshold change amount is determined based on the obtained one or more system parameters and wherein the one or more system parameters comprise parameters indicating available network bandwidth, storage space, and CPU processing capacity;
determining, by the computer system, whether differences between the base file and the revised version of the base file exceed the determined threshold change amount between the base file and the revised version of the base file; and
in response to determining the threshold change amount is exceeded, avoiding, by the computer system, completion of the delta file and transmitting a copy of the revised version of the base file for replication purposes.
2. The method of claim 1 , further comprising:
comparing, by the computer system, signatures of the base file and the revised version of the base file; and
shifting, by the computer system, a reference frame of data of a set length of the revised version of the base file and counting an amount of said shifting when the base file and the revised version of the base file do not match.
3. The method of claim 2 , wherein determining whether the differences between the base file and the revised version of the base file exceed the determined threshold change amount between the base file and the revised version of the base file further comprises comparing an amount of said shifting to a threshold to establish if the threshold change amount between the base file and the revised version of the base file has been exceeded.
4. The method of claim 3 , wherein determining whether the differences between the base file and the revised version of the base file exceed the determined threshold change amount between the base file and the revised version of the base file further comprises comparing an identifier for the shifted reference frame of data with an identifier for a block of data in the base file.
5. The method of claim 4 , wherein the identifier for the block of data in the base file comprises a cyclic redundancy check value residing in a base signature file.
6. A system comprising:
a processor; and
a memory coupled with and readable by the processor and storing therein a set of instructions which, when executed by the processor, causes the processor to determine whether to create a delta file reflecting differences between a base file and a revised version of the base file by:
obtaining one or more system parameters relating to resources of the computer system;
determining a threshold change amount between the base file and the revised version of the base file, wherein the threshold change amount is determined based on the obtained one or more system parameters and wherein the one or more system parameters comprise parameters indicating available network bandwidth, storage space, and CPU processing capacity;
determining whether differences between the base file and the revised version of the base file exceed the determined threshold change amount between the base file and the revised version of the base file; and
in response to determining the threshold change amount is exceeded, avoiding completion of the delta file and transmitting a copy of the revised version of the base file for replication purposes.
7. The system of claim 6 , further comprising:
comparing signatures of the base file and the revised version of the base file; and
shifting a reference frame of data of a set length of the revised version of the base file and counting an amount of said shifting when the base file and the revised version of the base file do not match.
8. The system of claim 7 , wherein determining whether the differences between the base file and the revised version of the base file exceed the determined threshold change amount between the base file and the revised version of the base file further comprises comparing an amount of said shifting to a threshold to establish if the threshold change amount between the base file and the revised version of the base file has been exceeded.
9. The system of claim 8 , wherein determining whether the differences between the base file and the revised version of the base file exceed the determined threshold change amount between the base file and the revised version of the base file further comprises comparing an identifier for the shifted reference frame of data with an identifier for a block of data in the base file.
10. The system of claim 9 , wherein the identifier for the block of data in the base file comprises a cyclic redundancy check value residing in a base signature file.
11. A computer-readable memory device comprising a set of instructions stored therein which, when executed by a processor, causes the processor to determine whether to create a delta file reflecting differences between a base file and a revised version of the base file by:
obtaining one or more system parameters relating to resources of the computer system;
determining a threshold change amount between the base file and the revised version of the base file, wherein the threshold change amount is determined based on the obtained one or more system parameters and wherein the one or more system parameters comprise parameters indicating available network bandwidth, storage space, and CPU processing capacity;
determining whether differences between the base file and the revised version of the base file exceed the determined threshold change amount between the base file and the revised version of the base file; and
in response to determining the threshold change amount is exceeded, avoiding completion of the delta file and transmitting a copy of the revised version of the base file for replication purposes.
12. The computer-readable memory device of claim 11 , further comprising:
comparing signatures of the base file and the revised version of the base file; and
shifting a reference frame of data of a set length of the revised version of the base file and counting an amount of said shifting when the base file and the revised version of the base file do not match.
13. The computer-readable memory device of claim 12 , wherein determining whether the differences between the base file and the revised version of the base file exceed the determined threshold change amount between the base file and the revised version of the base file further comprises comparing an amount of said shifting to a threshold to establish if the threshold change amount between the base file and the revised version of the base file has been exceeded.
14. The computer-readable memory device of claim 13 , wherein determining whether the differences between the base file and the revised version of the base file exceed the determined threshold change amount between the base file and the revised version of the base file further comprises comparing an identifier for the shifted reference frame of data with an identifier for a block of data in the base file.
15. The computer-readable memory device of claim 14 , wherein the identifier for the block of data in the base file comprises a cyclic redundancy check value residing in a base signature file.