Event based analytics database synchronization
Event based analytics database synchronization is described. In one example, an event tracking engine can sequence through batches of records in a transactional database that is being synchronized to an analytics database. The event tracking engine can determine a last version attribute associated with the batch, and query the transactional database to identify any records for the batch having a more recent version attribute than the last version attribute. When any records in the transactional database having a more recent version attribute are identified, the event tracking engine finds the data values in the record are different than those in the analytics database. The data values can be normalized, for example, before being forwarded to an analytics system for incorporation into the a analytics database. Over time, changes in the transactional database are identified and populated or mirrored in the analytics database.
1. A method for event based synchronization of data between databases, comprising:
enrolling a plurality of devices with a management service, the management service remotely configuring operating systems of the client devices to register and configure the client devices for device management;
sequencing, with reference to a tracking table, to a batch of records in a transactional database, the batch of records comprising a plurality of records corresponding to the plurality of devices;
determining, with reference to the tracking table, a last version attribute associated with data synchronization for the batch;
querying the transactional database to identify at least one record of the batch having a more recent version attribute than the last version attribute;
comparing data in an analytics database to data in the at least one record to find at least one value to synchronize from the transactional database to the analytics database; and
forwarding the at least one value to an analytics computing system for incorporation into an analytics database.
2. The method according to claim 1 , further comprising updating the last version attribute for the batch in the tracking table to reflect the more recent version attribute associated with the at least one record from the batch.
3. The method according to claim 1 , wherein the forwarding comprises:
normalizing the at least one value;
storing the at least one value in a staging table; and
forwarding the at least one value from the staging table to the analytics computing system for incorporation into the analytics database.
4. The method according to claim 1 , further comprising:
determining whether the transactional database comprises a record for a new device; and
in response the transactional database comprising the record for the new device, updating the tracking table to account for the new device.
5. The method according to claim 1 , wherein the sequencing comprises:
determining whether the tracking table identifies a new device; and
in response to the transactional database comprising the new device, selecting the batch as a next batch for processing based on the new device, wherein the batch encompasses a device identifier of the new device.
6. The method according to claim 1 , wherein the sequencing comprises:
reviewing a status of at least one batch synchronization process to identify at least one of a queued, in progress, completed, or error status for the least one batch synchronization process; and
selecting the batch as a next batch for processing based on the status.
7. The method according to claim 1 , wherein the sequencing comprises:
reviewing a timing associated with a status of at least one batch synchronization process; and
selecting the batch as a next batch for processing based on the timing and a frequency for batch synchronization processes.
8. The method according to claim 1 , further comprising importing data values from records in the transactional database to the analytics database.
9. The method according to claim 8 , wherein the importing comprises:
appending a version attribute to at least one record in the transactional database for the batch; and
updating the tracking table to incorporate the version attribute as the last version attribute associated with data synchronization for the batch.
10. A non-transitory computer-readable medium embodying program code executable in at least one computing device for event based synchronization of data between databases that, when executed by the at least one computing device, directs the at least one computing device to at least:
enroll a plurality of devices with a management service, the management service remotely configuring operating systems of the client devices to register and configure the client devices for device management;
sequence to a batch of records in a transactional database, the batch of records comprising a plurality of records corresponding to the plurality of devices;
determine a last version attribute associated with data synchronization for the batch;
query the transactional database to identify at least one record of the batch having a more recent version attribute than the last version attribute;
compare data in an analytics database to data in the at least one record to find at least one value to synchronize from the transactional database to the analytics database; and
forward the at least one value for incorporation into an analytics database.
11. The non-transitory computer-readable medium according to claim 10 , wherein the at least one computing device is further directed to at least update the last version attribute for the batch in a tracking table to reflect the more recent version attribute associated with the at least one record from the batch.
12. The non-transitory computer-readable medium according to claim 10 , wherein the at least one computing device is further directed to at least:
normalize the at least one value;
store the at least one value in a staging table; and
forward the at least one value from the staging table to an analytics computing system for incorporation into the analytics database.
13. The non-transitory computer-readable medium according to claim 10 , wherein the at least one computing device is further directed to at least:
determine whether the transactional database comprises a record for a new device; and
in response to the transactional database comprising the record for the new device, update a tracking table to account for the new device.
14. The non-transitory computer-readable medium according to claim 10 , wherein the at least one computing device is further directed to at least:
review a status of at least one batch synchronization process to identify at least one of a queued, in progress, completed, or error status for the least one batch synchronization process; and
select the batch as a next batch for processing based on the status.
15. A system for event based synchronization of data between databases, comprising:
a memory device configured to store computer-readable instructions thereon; and
at least one processing device configured, through execution of the computer-readable instructions, to at least:
enroll a plurality of devices with a management service, the management service remotely configuring operating systems of the client devices to register and configure the client devices for device management;
sequence to a batch of records in a transactional database, the batch of records comprising a plurality of records corresponding to the plurality of devices;
determine a last version attribute associated with data synchronization for the batch;
query the transactional database to identify at least one record for the batch having a more recent version attribute than the last version attribute; compare data in an analytics database to data in the at least one record to find at least one value to synchronize from the transactional database to the analytics database; and
forward the at least one value for incorporation into an analytics database.
16. The system according to claim 15 , wherein the at least one processing device is further configured to at least update the last version attribute for the batch in a tracking table to reflect the more recent version attribute associated with the at least one record from the batch.
17. The system according to claim 15 , wherein the at least one processing device is further configured to at least:
normalize the at least one value;
store the at least one value in a staging table; and
forward the at least one value from the staging table to an analytics computing system for incorporation into the analytics database.
18. The system according to claim 15 , wherein the at least one processing device is further configured to at least:
determine whether the transactional database comprises a record for a new device; and
in response to the transactional database comprising the record for the new device, update a tracking table to account for the new device.
19. The system according to claim 15 , wherein the at least one processing device is further configured to at least:
determine whether the transactional database comprises a record for a new device; and
in response to the transactional database comprising the new device, select the batch as a next batch for processing based on the new device, wherein the batch encompasses a device identifier of the new device.
20. The system according to claim 18 , wherein the at least one processing device is further configured to at least:
review a status of at least one batch synchronization process to identify at least one of a queued, in progress, completed, or error status for the least one batch synchronization process; and
select the batch as a next batch for processing based on the status.