Hybrid real-time data replication
View Patent ↗Techniques are described for replicating data from one to one or more heterogeneous data processing or communication devices for the purpose of remote backup, redundancy, content distribution, communications, observations or measurements. In a first phase, the attributes of the data that are modified or created on a device or that are passing through the device are tracked and journaled in volatile or non-volatile storage in real-time. In a second phase, the attributes that match patterns pre-specified in a configuration are used to determine which data to replicate and which modifications to make the devices. In a third phase, the data is replicated. In a preferred embodiment, the described techniques comprise an application that runs on a host device or is embedded in a logic or memory device. The described invention is designed to be transparent for system redundancy and error recovery processes such as error correction, re-transmission on links, and raid configuration.
1. A data replication method comprising:
synchronously capturing data modifications, the synchronously capturing comprising:
accepting a request from a client device to modify data;
performing the data modifications; and
adding data attributes of the modified data to a message queue;
asynchronously replicating the data modifications to one or more destination devices, the asynchronously replicating comprising:
saving the data attributes in a journal queue on a storage device;
retrieving the data attributes from the journal queue;
comparing the data attributes to configuration data for each of at least one pathway, each of the at least one pathways corresponding to a destination device, the configuration data comprising a replication policy; and
for each of the data attributes and each of the at least one pathway, inserting the data attribute into a corresponding one of at least one pathway journal if the data attribute matches the configuration data for the pathway.
2. The method of claim 1 , wherein accepting the request from a client device comprises accepting a request to forward data from the client device to another client device.
3. The method of claim 1 , further comprising:
fetching at least one of modified and new data attributes from a device that performed the data alteration;
adding the fetched data attributes to a journal message queue;
removing each message from the journal message queue; and
saving data attributes of the message if the attributes match specified patterns for a destination device, wherein the data attributes are saved on storage of the device that performed the data alteration.
4. The method of claim 3 , further comprising:
retrieving one or more data attributes from the storage device for a destination device if replication to the destination device has not been suspended; and
sending content described by the data attribute to the destination device if replication to the destination device has not been suspended.
5. The method of claim 4 , further comprising initializing replication to perform synchronization of the content of source and destination devices by reading a list of all data attributes on the source device and adding the list of the data attributes to the journal message queue for additional processing if the data attributes match the patterns of data attributes to be replicated.
6. The method of claim 1 , wherein the client device comprises a camera that captures the data.
7. A device for replication of data across a distributed computing system, the device comprising:
a pass-through component that synchronously intercepts data modifications, wherein the pass-through component:
accepts a request to modify data from a client device;
performs the data modifications; and
adds data attributes of the modified data to a message queue;
a data replication engine that asynchronously replicates the data modifications on one or more remote storage devices, wherein the data replication engine:
saves the data attributes in a journal queue on one or more of the storage devices;
retrieves the data attributes from the journal queue;
compares the data attributes against configuration data for each of at least one pathway, each of the at least one pathway corresponding to a destination device, the configuration data comprising a replication policy; and
for each data attribute and each of the at least one pathway, inserts the data attribute into a corresponding one of at least one pathway journal if the data attribute matches the configuration data for the pathway.
8. The device of claim 7 , wherein the data replication engine communicates the status of the operation to the client device that requested the change when the client device requests that the status be communicated.
9. The device of claim 8 , wherein the data replication engine comprises:
an input thread that retrieves one or more modification attributes from the pass-through component;
a journal queue that stores the retrieved attributes; and
a journal thread that retrieves the attributes from the journal queue and inserts each attribute into each of one or more pathways, wherein each of the pathways comprises:
a pathway journal that stores the inserted attributes;
a remote thread to retrieve the attributes stored in the pathway journal and pass each of the attributes to a transport thread that replicates data to a represented by the attribute to a destination device;
a work journal that stores the attributes retrieved by the remote thread if the attributes are not presently stored in the work journal; and
a complete thread that removes each attributes stored in the work journal and the pathway journal if a reference count for the attribute is zero.
10. The device of claim 7 , wherein the data replication engine captures the data changes synchronously by saving the data attributes, and performs data replication asynchronously.
11. The device of claim 7 , wherein the data replication engine includes a transport thread to replicate the data to a destination device.