Optimized and consistent replication of file overwrites
In general, certain embodiments of the present disclosure provide techniques or mechanisms for creating efficient and consistent replication of file overwrites. According to various embodiments, a file map with a sequence file number is created for a data file on a source. The file map includes one or more nodes that each include a sequence node number. When an entry in the file map is changed, the sequence file number is incremented, and the corresponding node is updated by, among other things, setting the sequence node number to equal the current sequence file number. All other nodes in the file map that point to the updated node are similarly updated. When replication occurs, only the updated nodes in the file map are replicated and any unreplicated non-updated nodes are maintained.
1. A method comprising:
creating a file map to track data chunks and at least one pointer to a data file, wherein the file map includes one or more nodes, each node including a sequence node number;
changing an entry associated with a data chunk of the data file in the file map, the entry corresponding to a node in the file map, the data chunk being stored at the node;
incrementing a previous sequence file number for the file map to form a current sequence file number for the file map;
updating the node corresponding to the changed entry, wherein updating the node includes setting the sequence node number of the node to equal the current sequence file number;
updating each node that includes a pointer to the data chunk being stored at the node, wherein updating each node includes setting the sequence node number of that particular node to the current sequence file number;
replicating the updated nodes in the file map and refraining from replicating non-updated nodes, wherein a respective sequence node number of the non-updated nodes are not equal to the current sequence file number, wherein a first non-updated node has a first sequence node number, wherein a second non-updated node has the first sequence node number, wherein the first sequence node number is not equal to the current sequence file number;
connecting new pointers from the replicated updated nodes to the non-updated nodes to which the original versions of the updated nodes had pointers.
2. The method of claim 1 , wherein the file map is represented as a tree data structure.
3. The method of claim 1 , wherein only the updated nodes with a sequence node number greater than a replication sequence number are replicated, wherein the replication sequence number corresponds to the number of successful replications at a target.
4. The method of claim 3 , further comprising updating the replication sequence number only after completion of a successful replication at the target, wherein updating the replication sequence number includes setting the replication sequence number to equal the current sequence file number upon successful replication of the data file at the target.
5. The method of claim 1 , wherein a read only copy of the file map is maintained at the target for read requests during replication, the read only copy corresponding to the version of the file map immediately before replication begins.
6. The method of claim 1 , wherein changing the entry corresponds to changing a number of file map pages in the file map.
7. The method of claim 6 , wherein updates to reference counts utilized in data deduplication are proportional to the number of file map pages that have been changed.
8. A system, comprising:
a processor;
memory, wherein the memory contains instructions to cause the processor to:
create a file map to track data chunks and at least one pointer to a data file, wherein the file map includes one or more nodes, each node including a sequence node number;
change an entry associated with a data chunk of the data file in the file map, the entry corresponding to a node in the file map, the data chunk being stored at the node;
increment a previous sequence file number for the file map to form a current sequence file number for the file map;
update the node corresponding to the changed entry, wherein updating the node includes setting the sequence node number of the node to equal the current sequence file number;
update each node that includes a pointer to the data chunk being stored at the node, wherein updating each node includes setting the sequence node number of that particular node to the current sequence file number;
replicate the updated nodes in the file map and refrain from replicating non-updated nodes, wherein a respective sequence node number of the non-updated nodes are not equal to the current sequence file number, wherein a first non-updated node has a first sequence node number, wherein a second non-updated node has the first sequence node number, wherein the first sequence node number is not equal to the current sequence file number;
connect new pointers from the replicated updated nodes to the non-updated nodes to which the original versions of the updated nodes had pointers.
9. The system of claim 8 , wherein the file map is represented as a tree data structure.
10. The system of claim 8 , wherein only the updated nodes with a sequence node number greater than a replication sequence number are replicated, wherein the replication sequence number corresponds to the number of successful replications at a target.
11. The system of claim 10 , further comprising updating the replication sequence number only after completion of a successful replication at the target, wherein updating the replication sequence number includes setting the replication sequence number to equal the current sequence file number upon successful replication of the data file at the target.
12. The system of claim 8 , wherein a read only copy of the file map is maintained at the target for read requests during replication, the read only copy corresponding to the version of the file map immediately before replication begins.
13. The system of claim 8 , wherein changing the entry corresponds to changing a number of file map pages in the file map.
14. The system of claim 13 , wherein updates to reference counts utilized in data deduplication are proportional to the number of file map pages that have been changed.
15. A non-transitory computer readable medium containing computer code that, when executed by one or more processors, cause a system to perform operations comprising:
creating a file map to track data chunks and at least one pointer to a data file, wherein the file map includes one or more nodes, each node including a sequence node number;
changing an entry associated with a data chunk of the data file in the file map, the entry corresponding to a node in the file map, the data chunk being stored at the node;
incrementing a previous sequence file number for the file map to form a current sequence file number for the file map;
updating the node corresponding to the changed entry, wherein updating the node includes setting the sequence node number of the node to equal the current sequence file number;
updating each node that includes a pointer to the data chunk being stored at the node, wherein updating each node includes setting the sequence node number of that particular node to the current sequence file number;
replicating the updated nodes in the file map and refraining from replicating non-updated nodes, wherein a respective sequence node number of the non-updated nodes are not equal to the current sequence file number, wherein a first non-updated node has a first sequence node number, wherein a second non-updated node has the first sequence node number, wherein the first sequence node number is not equal to the current sequence file number; and
connecting new pointers from the replicated updated nodes to the non-updated nodes to which the original versions of the updated nodes had pointers.
16. The non-transitory computer readable medium of claim 15 , wherein the file map is represented as a tree data structure.
17. The non-transitory computer readable medium of claim 15 , wherein only the updated nodes with a sequence node number greater than a replication sequence number are replicated, wherein the replication sequence number corresponds to the number of successful replications at a target.
18. The non-transitory computer readable medium of claim 17 , further comprising updating the replication sequence number only after completion of a successful replication at the target, wherein updating the replication sequence number includes setting the replication sequence number to equal the current sequence file number upon successful replication of the data file at the target.
19. The non-transitory computer readable medium of claim 15 , wherein changing the entry corresponds to changing a number of file map pages in the file map.
20. The non-transitory computer readable medium of claim 15 , wherein a read only copy of the file map is maintained at the target for read requests during replication, the read only copy corresponding to the version of the file map immediately before replication begins.