Systems and methods for performing data replication
View Patent ↗Preparing source data to be replicated in a continuous data replication environment. Certain systems and methods populate a file name database with entries having a unique file identifier descriptor (FID), short name and a FID of the parent directory of each directory or file on a source storage device. Such information is advantageously gathered during scanning of a live file system without requiring a snapshot of the source storage device. The database can be further used to generate absolute file names associated with data operations to be replayed on a destination storage device. Based on the obtained FIDs, certain embodiments can further combine write operations to be replayed on the destination storage device and/or avoid replicating temporary files to the destination system.
1. A method for performing data replication, the method comprising:
monitoring with one or more processors of a source storage system a plurality of journal entries associated with writing data to a source storage device associated with the source storage system;
identifying a first journal entry of the plurality of journal entries, the first journal entry comprising a first data write operation, a first file identifier descriptor (FID) of a file to be modified by the first data write operation on the source storage device, and a first location of a first portion of the file to be modified,
wherein a FID identifies a file or directory of a file system on the source storage system and is usable to construct an absolute file name for transmitting data to a replication system;
identifying a second journal entry of the plurality of journal entries, the second journal entry comprising a second data write operation, a second FID of a file to be modified by the second data write operation on the source storage device, and a second location of a second portion of the file to be modified,
wherein the first journal entry and the second journal entry are from the same journal;
determining with the one or more processors that the first and second data write operations can be combined into a single write operation based on a determination that the first and second FIDs both correspond to a first value;
combining the first and second data write operations based on said determination;
constructing with the one or more processors an absolute file name by associating the first value with a short name and at least one directory name, wherein neither the first nor second journal entries comprises the absolute file name; and
transmitting the single write operation and the absolute file name to a destination storage device to replay on the destination storage device the data modifications associated with the first and second write operations,
wherein the destination storage device stores a replicated version of data written to the source storage device.
2. The method of claim 1 , wherein the first location comprises a first byte range and the second location comprises a second byte range.
3. The method of claim 2 , wherein said determining that the first and second data write operations can be combined further comprises analyzing a distance between the first and second byte ranges.
4. The method of claim 3 , wherein determining that the first and second data write operations can be combined further comprises determining that the distance is less than a threshold amount.
5. The method of claim 4 , wherein the threshold amount is less than 1 KB.
6. The method of claim 5 , wherein the threshold amount is approximately 200 bytes.
7. The method of claim 1 , wherein said monitoring operations comprises processing the plurality of journal entries with a filter driver.
8. The method of claim 1 , wherein said monitoring comprises analyzing a portion of the plurality of journal entries as received over a predetermined period of time.
9. The method of claim 8 , additionally comprising adjusting the predetermined period of time based on a known good state of the source storage device data.
10. The method of claim 8 , wherein said predetermined period of time is approximately four seconds.
11. The method of claim 1 , additionally comprising:
identifying multiple ones of the plurality of journal entries having other write operations that cannot be combined;
reading byte ranges associated with the other write operations in order of increasing file offsets; and
transmitting the other write operations to the destination storage device for replay on the destination storage device in the order of the increasing file offsets.
12. A system for performing data replication, the system comprising:
at least one computer application executing on a computing device and configured to generate operations associated with data on a source storage device associated with a source storage system;
a filter module disposed between the at least one computer application and the source storage device, the filter module configured to:
monitor a plurality of journal entries associated with writing data to the source storage device;
identify a first journal entry of the plurality of journal entries, the first journal entry comprising a first data modification operation, a first file identifier descriptor (FID) of a file to be modified by the first data modification operation, and a first location of a first portion of the file to be modified,
wherein a FID identifies a file or directory of a file system on the source storage system and is usable to construct an absolute file name for transmitting data to a replication system;
identify a second journal entry of the plurality of journal entries, the second journal entry comprising a second data modification operation, a second FID of a file to be modified by the second data modification operation, and a second location of a second portion of the file to be modified,
wherein the first journal entry and the second journal entry are from the same journal;
a processing module configured to:
determine that the first and second data modification operations can be combined into a single modification operation based on a determination that the first and second FIDs both correspond to a first value; and
combine the first and second data modification operations based on said determination; and
at least one database thread configured to construct an absolute file name for replaying the single modification operation on replication data of a destination storage device by associating the first value with a short name and at least one directory name, wherein neither the first nor second journal entries comprises the absolute file name,
wherein the destination storage device stores a replicated version of data written to the source storage device.
13. The system of claim 12 , wherein the filter module comprises a file system filter driver.
14. The system of claim 13 , wherein the first data modification operation comprises a write operation, and wherein the file system filter driver is configured to obtain the first data modification operation and the first FID without receiving a copy of data to be written to the source storage device.
15. The system of claim 12 , further comprising a database comprising file name data that associates each of the first and second FIDs with a short name and a parent FID.
16. The system of claim 12 , wherein the first file location comprises an offset and a length of a byte range of the file to be modified.
17. The system of claim 12 , further comprising a memory configured to store a portion of the operations generated over a predetermined period of time.
18. The system of claim 17 , wherein the predetermined period of time is at least three seconds.
19. A system for performing data replication, the system comprising:
means for monitoring a plurality of journal entries associated with writing data to a source storage device associated with a source storage system;
means for identifying a first journal entry of the plurality of journal entries, the first journal entry comprising a first data write operation, a first file identifier descriptor (FID) of a file to be modified on the source storage device, and a first location of a first portion of the file to be modified, and for identifying a second journal entry of the plurality of journal entries, the second journal entry comprising a second data write operation, a second FID of a file to be modified on the source storage device, and a second location of a second portion of the file to be modified,
wherein a FID identifies a file or directory of a file system on the source storage system and is usable to construct an absolute file name for transmitting data to a replication system, and
wherein the first journal entry and the second journal entry are from the same journal;
means for determining that the first and second data write operations can be combined into a single write operation based on a determination that the first and second FIDs both correspond to a first value, and for combining the first and second data write operations based on said determination;
means for constructing an absolute file name by associating the first value with a short name and at least one directory name, wherein neither the first nor second journal entries comprises the absolute file name; and
means for transmitting the single write operation and the absolute file name to a destination storage device to replay on the destination storage device the data modifications associated with the first and second write operations,
wherein the destination storage device stores a replicated version of data written to the source storage device.
20. The system of claim 19 , further comprising means for storing file name data that associates each of the first and second FIDs with a short name and a parent FID usable to construct the absolute file name.