Efficient differential techniques for metafiles
Exemplary embodiments provide methods, mediums, and systems for replicating metafiles between a source and a destination. The metafile may be subdivided into blocks. The contents of the metafile may be transferred by locating the blocks which are changed between the source version of the metafile and the destination version of the metafile. The changed blocks may be examined to retrieve the contents of the changed blocks. The records in the changed blocks may be evaluated to determine whether to create a corresponding record at the destination, delete a corresponding record at the destination, or update a corresponding record at the destination. Accordingly, the metafile may be replicated in a logical manner, by transferring only changed records rather than the entirety of a changed block. Moreover, the transfer is conducted efficiently because unchanged blocks are eliminated from consideration at the outset.
1. A method comprising:
identifying a changed block different between a first file at a first file system and a second file at a second file system;
extracting a first set of records of the changed block in the first file and a second set of records of the changed block in the second file;
identifying a set of changed records and a set of unchanged records of the changed block by:
comparing key/value pairs of records within the first set of records to key/value pairs of corresponding records within the second set of records, wherein a key is an identifier of file data and a value is location information of the file data;
identifying the set of changed records as records and corresponding records that have non-matching key/value pairs; and
identifying the set of changed records as records and corresponding records that have matching key/value pairs;
transmitting the set of changed records of the changed block in the first file to the second file system for updating the changed block in the second file, wherein transmission skips transmitting the set of unchanged records in the changed block.
2. The method of claim 1 , comprising:
utilizing a buff tree to identify the changed block.
3. The method of claim 1 , wherein the first file comprises a metafile and the second file comprises a replicated version of the metafile.
4. The method of claim 1 , wherein the key comprises a name of the first file.
5. The method of claim 1 , wherein the key comprises a hash value calculated from a name of the first file.
6. The method of claim 1 , wherein the value comprises an absolute location of the first file in a storage volume.
7. The method of claim 1 , wherein the value comprises an offset of the first file.
8. The method of claim 1 , wherein records stored within the first file are stored in non-fixed locations.
9. The method of claim 1 , wherein the first file is represented by a B+ tree.
10. The method of claim 1 , wherein the first file is stored by the first file system according to a format that is un-readable by the second file system.
11. The method of claim 1 , wherein the first file is represented by a snapshot.
12. A non-transitory machine readable medium comprising instructions for performing a method, which when executed by a machine, causes the machine to:
identify a changed block different between a first file at a first file system and a second file at a second file system;
extract a first set of records of the changed block in the first file and a second set of records of the changed block in the second file;
identify a set of changed records and a set of unchanged records of the changed block by:
comparing key/value pairs of records within the first set of records to key/value pairs of corresponding records within the second set of records, wherein a key is an identifier of file data and a value is location information of the file data;
identifying the set of changed records as records and corresponding records that have non-matching key/value pairs; and
identifying the set of changed records as records and corresponding records that have matching key/value pairs;
transmit the set of changed records of the changed block in the first file to the second file system for updating the changed block in the second file, wherein transmission skips transmitting the set of unchanged records in the changed block.
13. The non-transitory machine readable medium of claim 12 , wherein records stored within the first file are stored in non-fixed locations.
14. The non-transitory machine readable medium of claim 12 , wherein the first file is represented by a B+ tree.
15. The non-transitory machine readable medium of claim 12 , wherein the first file is stored by the first file system according to a format that is un-readable by the second file system.
16. The non-transitory machine readable medium of claim 12 , wherein the first file is represented by a snapshot.
17. A computing device comprising:
a memory containing machine readable medium comprising machine executable code having stored thereon instructions for performing a method; and
a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to:
identify a changed block different between a first file at a first file system and a second file at a second file system;
extract a first set of records of the changed block in the first file and a second set of records of the changed block in the second file;
identify a set of changed records and a set of unchanged records of the changed block by:
comparing key/value pairs of records within the first set of records to key/value pairs of corresponding records within the second set of records, wherein a key is an identifier of file data and a value is location information of the file data;
identifying the set of changed records as records and corresponding records that have non-matching key/value pairs; and
identifying the set of unchanged records as records and corresponding records that have matching key/value pairs;
transmit the set of changed records of the changed block in the first file to the second file system for updating the changed block in the second file, wherein transmission skips transmitting the set of unchanged records in the changed block.
18. The computing device of claim 17 , wherein the first file comprises a metafile and the second file comprises a replicated version of the metafile.
19. The computing device of claim 17 , wherein the key comprises a name of the first file.
20. The computing device of claim 17 , wherein the key comprises a hash value calculated from a name of the first file.