Identifying changes between source and destination databases using snapshots of source database and invertible bloom filters
A centralized database management system performs data synchronization with lower bandwidth consumption and higher efficiency. The system manages data synchronization and data reconciliation across multiple databases managed by multiple DBMS across different client servers. The system generates and sends instructions that encode each data table into an invertible bloom filter and identifies differences between the two databases by performing a subtraction operation on the two invertible bloom filters. The system may generate a third invertible bloom filter comprising information associated with differences between the two data tables. The system may send instructions to the source and the destination databases, where a first and a second invertible bloom filters are encoded for the source and the destination databases, respectively. The system may decode the third invertible bloom filter, identify the different elements, and generate instructions to the source and/or the destination database.
1. A method comprising:
retrieving a first snapshot of a source data table, the first snapshot including information of the source data table captured at a first point in time;
retrieving a second snapshot of the source data table, the second snapshot including information of the source data table at a second point in time later than the first point in time;
determining a size for both a first invertible bloom filter and a second invertible bloom filter based on an estimated number of changes to the source data table over an interval of time between the first point in time and the second point in time,
wherein determining the size for both the first invertible bloom filter and the second invertible bloom filter comprises:
determining the size as a constant that is larger than a threshold; and
decreasing the size responsive to a number of changes to the source data table being smaller than the constant;
obtaining the first invertible bloom filter for the source data table based on the first snapshot of the source data table, the first invertible bloom filter having the determined size;
obtaining the second invertible bloom filter for the source data table based on the second snapshot of the source data table, the second invertible bloom filter having the determined size;
determining that a destination data table has outdated information relative to that of the first point in time by:
generating a third invertible bloom filter by subtracting the second invertible bloom filter from the first invertible bloom filter, the third invertible bloom filter comprising information associated with a change between the first snapshot and the second snapshot;
identifying the change by decoding the third invertible bloom filter; and
transmitting information associated with the identified change to the destination data table to synchronize the destination data table with the source data table based on the identified change.
2. The method of claim 1 , further comprising responsive to the first and the second invertible bloom filter including a number of empty cells, compressing the first invertible bloom filter into a compressed invertible bloom filter by reducing the size.
3. The method of claim 1 , further comprising generating a backup invertible bloom filter for the first invertible bloom filter, wherein the backup invertible bloom filter is activated responsive to a number of changes between the first snapshot and the second snapshot being greater than a predetermined threshold.
4. The method of claim 1 , wherein the change to the source data table is identified without accessing data entries of the data table.
5. A non-transitory computer-readable storage medium storing executable computer instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the instructions comprising instructions to:
retrieve a first snapshot of a source data table, the first snapshot including information of the source data table captured at a first point in time;
retrieve a second snapshot of the source data table, the second snapshot including information of the source data table at a second point in time later than the first point in time;
determine a size for both a first invertible bloom filter and a second invertible bloom filter based on an estimated number of changes to the source data table over an interval of time between the first point in time and the second point in time,
wherein determining the size for both the first invertible bloom filter and the second invertible bloom filter comprises:
determining the size as a constant that is larger than a threshold; and
decreasing the size responsive to a number of changes to the source data table being smaller than the constant;
obtain the first invertible bloom filter for the source data table based on the first snapshot of the source data table, the first invertible bloom filter having the determined size;
obtain the second invertible bloom filter for the source data table based on the second snapshot of the source data table, the second invertible bloom filter having the determined size;
determine that a destination data table has outdated information relative to that of the first point in time by:
generating a third invertible bloom filter by subtracting the second invertible bloom filter from the first invertible bloom filter, the third invertible bloom filter comprising information associated with a change between the first snapshot and the second snapshot;
identify the change by decoding the third invertible bloom filter; and
transmit information associated with the identified change to the destination data table to synchronize the destination data table with the source data table based on the identified change.
6. The non-transitory computer-readable storage medium of claim 5 , wherein the instructions further comprise instructions to, responsive to the first and the second invertible bloom filter including a number of empty cells, compress the first invertible bloom filter into a compressed invertible bloom filter by reducing the size.
7. The non-transitory computer-readable storage medium of claim 5 , wherein the instructions further comprise instructions to generate a backup invertible bloom filter for the first invertible bloom filter, wherein the backup invertible bloom filter is activated responsive to a number of changes between the first snapshot and the second snapshot being greater than a predetermined threshold.
8. The non-transitory computer-readable storage medium of claim 5 , wherein the change to the source data table is identified without accessing data entries of the data table.
9. A computing system comprising:
a processor; and
a non-transitory computer-readable storage medium storing instructions for database synchronization, the instructions when executed by the processor cause the processor to perform steps including:
retrieving a first snapshot of a source data table, the first snapshot including information of the source data table captured at a first point in time;
retrieving a second snapshot of the source data table, the second snapshot including information of the source data table at a second point in time later than the first point in time;
determining a size for both a first invertible bloom filter and a second invertible bloom filter based on an estimated number of changes to the source data table over an interval of time between the first point in time and the second point in time,
wherein determining the size for both the first invertible bloom filter and the second invertible bloom filter comprises:
determining the size as a constant that is larger than a threshold; and
decreasing the size responsive to a number of changes to the source data table being smaller than the constant;
obtaining the first invertible bloom filter for the source data table based on the first snapshot of the source data table, the first invertible bloom filter having the determined size;
obtaining the second invertible bloom filter for the source data table based on the second snapshot of the source data table, the second invertible bloom filter having the determined size;
determining that a destination data table has outdated information relative to that of the first point in time by:
generating a third invertible bloom filter by subtracting the second invertible bloom filter from the first invertible bloom filter, the third invertible bloom filter comprising information associated with a change between the first snapshot and the second snapshot;
identifying the change by decoding the third invertible bloom filter; and
transmitting information associated with the identified change to the destination data table to synchronize the destination data table with the source data table based on the identified change.
10. The computing system of claim 9 , wherein the steps further comprise: generating a resizable invertible bloom filter for the first snapshot, wherein the resizable invertible bloom filter is initialized with a maximum size and is shrunken to a second size smaller than the maximum size, wherein the second size is selected from a set of predetermined sizes.
11. The method of claim 1 , further comprising:
detecting that a number of changes to the source data table is different from the estimated number of changes to the source data table over the interval of time; and
adjusting the size for both invertible bloom filters based on the detected number of changes to the source data table.
12. The method of claim 1 , wherein determining the size for both the first invertible bloom filter and the second invertible bloom filter comprises:
using a machine learning model to predict the estimated number of changes, wherein the machine learning model is trained based on features extracted from historically observed changes to the source data table over the interval of time.
13. The non-transitory computer-readable storage medium of claim 5 , wherein the instructions further comprise instructions to:
detect that a number of changes to the source data table is different from the estimated number of changes to the source data table over the interval of time; and
adjust the size for both invertible bloom filters based on the detected number of changes to the source data table.
14. The non-transitory computer-readable storage medium of claim 5 , wherein the instructions to determine the size for both the first invertible bloom filter and the second invertible bloom filter comprise:
using a machine learning model to predict the estimated number of changes, wherein the machine learning model is trained based on features extracted from historically observed changes to the source data table over the interval of time.
15. The computing system of claim 9 , wherein the steps further comprise:
detecting that a number of changes to the source data table is different from the estimated number of changes to the source data table over the interval of time; and
adjusting the size for both invertible bloom filters based on the detected number of changes to the source data table.
16. The computing system of claim 9 , wherein determining the size for both the first invertible bloom filter and the second invertible bloom filter comprises:
using a machine learning model to predict the estimated number of changes, wherein the machine learning model is trained based on features extracted from historically observed changes to the source data table over the interval of time.