System and method for streaming data conversion and replication
View Patent ↗A system and method are provided for integrating conversion and replication of a data stream for replicating data stored in a primary data store. The method includes streaming a replicated data stream from the primary data store to a secondary data store. The method includes intercepting the replicated data stream in route to the secondary data store. The method also provides for converting the replicated data stream.
1. A data streaming conversion and replication system, comprising:
a primary data store that stores data according to a first format;
a secondary data store that maintains a back-up of the data stored in the primary data store, the back-up in the secondary data store stored according to a second format;
a network connection coupling the primary data store and the secondary data store;
a replicator that automatically replicates data between the primary data store and the secondary data store as changes are made to the data stored in the primary data store;
a conversion unit that converts the data from the first format to the second format, wherein the conversion unit is transparent to the replicator, the primary data store, and the secondary data store, and wherein the conversion unit is transparent by passively converting the data without interacting with the replicator, the primary data store, and the secondary data store; and
the replicator and the conversion unit replicate and convert the data substantially streamed between the primary data store and the secondary data store.
2. The system of claim 1 , the replicator performs real-time asynchronous replication.
3. The system of claim 1 , wherein the conversion unit performs byte reversal.
4. The system of claim 1 , wherein the conversion unit performs byte size conversion.
5. The system of claim 1 , wherein the conversion unit performs character set conversion.
6. The system of claim 1 , the replicator further comprising replication software hosted at the primary data store and the secondary data store.
7. The system of claim 6 , the replicator further comprising one or more storage resource logs, wherein changes to data stored in the primary data store are written to a first storage resource log, streamed to a second storage resource log and stored in the secondary data store.
8. A conversion unit, comprising:
a processor;
a memory storing a conversion algorithm;
an inbound network connection streaming data in from a first data store;
an outbound network connection streaming data out to a second data store;
wherein the conversion algorithm, when invoked, causes the processor to:
intercept a replicated data stream at the inbound network connection;
convert the replicated data stream; and
resume the replicated data stream, as converted, at the outbound network connection,
wherein the conversion unit is transparent to any functional component to which it is connected via the inbound network connection or the outbound network connection; and
wherein the conversion unit is transparent by passively converting the replicated data stream without interacting with any functional component to which it is connected via the inbound network connection or the outbound network connection.
9. The conversion unit of claim 8 , wherein the conversion algorithm, when invoked, causes the processor to convert the replicated data stream, byte-by-byte, from one endianess format to the other.
10. The conversion unit of claim 8 , wherein the conversion algorithm, when invoked, causes the processor to convert the replicated data stream, byte-by-byte, from one character set to another.
11. The conversion unit of claim 8 , wherein the conversion algorithm, when invoked, causes the processor to convert the replicated data stream, byte-by-byte, from one byte size format to another.
12. A method for conversion and replication of a data stream, comprising:
adding or changing data in a primary data store via writes to the primary data store;
replicating each of the writes to the primary data store byte-by-byte as the data in the primary data store is added or changed;
streaming the replicated writes in a replicated data stream;
receiving the replicated data stream in route from the primary data store to a secondary data store;
converting the received replicated data stream byte-by-byte from a first format to a second format;
streaming the converted replicated data stream to the secondary data store; and
storing the replicated, received, and converted data to the secondary data store,
wherein the replicating, receiving, converting and storing occur while substantially streaming the data from the primary data store to the secondary data store, and
wherein receiving the replicated data stream includes intercepting the replicated data stream, wherein intercepting the replicated data stream is transparent to the primary data store and the secondary data store, and wherein intercepting the replicated data stream is transparent by passively converting the replicated data stream without interacting with the primary data store and the secondary data store.
13. The method of claim 12 , wherein replicating data further comprises real-time asynchronous replication.
14. The method of claim 12 , wherein converting the replicated data stream further comprises one or more of performing byte reversal, performing byte size conversion, and performing character set conversion.
15. The method of claim 12 , wherein replicating each of the writes minimally induces loss or significant reduction in access to the primary data store.