Apparatus and method for guaranteeing exactly once delivery of change data events to a document-oriented database on recovery
A non-transitory computer readable storage medium has instructions executed by a processor to receive change data events from a data source. Ordered positions of the change data events are maintained using a system checkpoint. The change data events are written to a document-oriented data store including a change data event collection and a change data event target checkpoint collection. There is an identification when an ordered position of change data events at the system checkpoint is lower than an ordered position of change data events in the change data event target checkpoint collection. Data source records following a position of change data events at the system checkpoint are collected. Records up to the position of change data events in the change data event target checkpoint collection are discarded and de-duplicated for exactly once delivery.
1 . A non-transitory computer readable storage medium with instructions executed by a processor to:
receive at a data replication module change data events from data sources, where the data replication module is executed by a server in network communication with the data sources;
maintain ordered positions of the change data events using a system checkpoint within the data replication module, where each particular change data event has a unique position value that represents a data source origin of the particular change data event;
write the change data events to a document-oriented data store including a change data event collection and a change data event target checkpoint collection, where the document-oriented data store is in network communication with the server;
identify a discrepancy between an ordered position of change data events at the system checkpoint and an ordered position of change data events in the change data event target checkpoint collection, where the discrepancy indicates a system failure;
after the system failure, collect from the data source origin of the event, data source records that have been written to the document-oriented data store and data source records that have not been written to the document-oriented data store;
de-duplicate the data source records that have been written to the document-oriented data store; and
send to the document-oriented data store only the data source records that have not been written to the document-oriented data store.
2 . The non-transitory computer readable storage medium of claim 1 wherein the data sources include a network connected data source machine.
3 . The non-transitory computer readable storage medium of claim 2 wherein the network connected data source machine is a change data event stream machine.
4 . The non-transitory computer readable storage medium of claim 2 wherein the network connected data source machine is an operations log machine.
5 . The non-transitory computer readable storage medium of claim 2 wherein the network connected data source machine is a redo log machine.
6 . The non-transitory computer readable storage medium of claim 1 further comprising instructions executed by the processor to utilize a signature of the data replication module to support a bi-directional replication system between two document-oriented data stores without looping.