Efficient state synchronization in a clustered environment using compacted key/tuple representations and snapshot-based state restoration
Disclosed are some implementations of systems, apparatus, methods and computer program products for synchronizing data. A source device processes an update to data in a database. The source device transmits, via a message bus, a first event message pertaining to the update, the first event message having an associated indicator. A target device accessing the message bus detects the indicator. Responsive to detecting the indicator, the target device skips the first event message on the message bus and identifies a snapshot link in a second event message subsequent to the first event message. The target device accesses a snapshot event identified by the snapshot link, stores data of the snapshot event, and processes one or more event messages subsequent to the snapshot event.
1 . A method, comprising:
detecting, by a target device accessing a message bus, an indicator of a first event message transmitted by a source device via the message bus, the first event message pertaining to an update to data in a database, the update being processed by the source device;
responsive to detecting the indicator, skipping, by the target device, the first event message on the message bus;
listening, by the target device, within the message bus for a subsequent event message containing a snapshot link;
identifying, by the target device, the snapshot link in a second event message subsequent to the first event message on the message bus, the snapshot link pointing to a position in the message bus, the indicator indicating that the source device is unaware of the snapshot link;
resubscribing, by the target device, at the position in the message bus;
accessing, by the target device, a snapshot event identified by the snapshot link, the snapshot event including a complete snapshot of data from which subsequent event messages can be processed;
storing, by the target device, data of the snapshot event in an in-memory data structure;
processing, by the target device, one or more event messages subsequent to the snapshot event; and
updating the in-memory data structure according to the one or more event messages.
2 . The method of claim 1 , further comprising:
responsive to detecting a second snapshot link in one of the event messages, discontinuing processing the one or more event messages.
3 . The method of claim 1 , further comprising:
constructing, by the source device, a current image of the data represented by event messages of the message bus.
4 . The method of claim 3 , wherein constructing is performed simultaneous with transmitting the first event message.
5 . A computer program product comprising computer-readable program code capable of being executed by one or more processors when retrieved from a non-transitory computer-readable medium, the program code comprising computer-readable instructions configurable to cause:
detecting, by a target device accessing a message bus, an indicator of a first event message transmitted by a source device via the message bus, the first event message pertaining to an update to data in a database, the update being processed by the source device;
responsive to detecting the indicator, skipping, by the target device, the first event message on the message bus;
listening, by the target device, within the message bus for a subsequent event message containing a snapshot link;
identifying, by the target device, the snapshot link in a second event message subsequent to the first event message on the message bus, the snapshot link pointing to a position in the message bus, the indicator indicating that the source device is unaware of the snapshot link;
resubscribing, by the target device, at the position in the message bus;
accessing, by the target device, a snapshot event identified by the snapshot link, the snapshot event including a complete snapshot of data from which subsequent event messages can be processed;
storing, by the target device, data of the snapshot event in an in-memory data structure;
processing, by the target device, one or more event messages subsequent to the snapshot event; and
updating the in-memory data structure according to the one or more event messages.
6 . The computer program product of claim 5 , further comprising:
responsive to detecting a second snapshot link in one of the event messages, discontinuing processing the one or more event messages.
7 . The computer program product of claim 5 , further comprising:
constructing, by the source device, a current image of the data represented by event messages of the message bus.
8 . The computer program product of claim 7 , wherein constructing is performed simultaneous with transmitting the first event message.
9 . A computing system, comprising:
a processor; and
a memory, the computing system configurable to cause:
detecting, by a target device accessing a message bus, an indicator of a first event message transmitted by a source device via the message bus, the first event message pertaining to an update to data in a database, the update being processed by the source device;
responsive to detecting the indicator, skipping, by the target device, the first event message on the message bus;
listening, by the target device, within the message bus for a subsequent event message containing a snapshot link;
identifying, by the target device, the snapshot link in a second event message subsequent to the first event message on the message bus, the snapshot link pointing to a position in the message bus, the indicator indicating that the source device is unaware of the snapshot link;
resubscribing, by the target device, at the position in the message bus;
accessing, by the target device, a snapshot event identified by the snapshot link, the snapshot event including a complete snapshot of data from which subsequent event messages can be processed;
storing, by the target device, data of the snapshot event in an in-memory data structure;
processing, by the target device, one or more event messages subsequent to the snapshot event; and
updating the in-memory data structure according to the one or more event messages.
10 . The computing system of claim 9 , further comprising:
responsive to detecting a second snapshot link in one of the event messages, discontinuing processing the one or more event messages.
11 . The computing system of claim 9 , further comprising:
constructing, by the source device, a current image of the data represented by event messages of the message bus.
12 . The computing system of claim 11 , wherein constructing is performed simultaneous with transmitting the first event message.
13 . The method of claim 1 , the indicator indicating that the source device is unaware of a most recent snapshot link.
14 . The method of claim 1 , the one or more event messages including one or more of: a CREATE, UPDATE, or DELETE operation.
15 . The method of claim 1 , wherein updating includes compacting the one or more event messages into a compacted representation and storing the compacted representation in the in-memory data structure.
16 . The method of claim 1 , the data of the snapshot event including multiple key-value entries.
17 . The method of claim 1 , wherein the source device and target device are both associated with a particular tenant in a multi-tenant database system.
18 . The method of claim 1 , the first event message including a key and a value.