Real-time data validation using lagging replica databases
A real-time data validation system prevents problematic updates from being replicated from a primary database to a replica database. The data validation system may inject a time delay into the replication process, so that when an update in a replication stream fails a specified data validation, the data validation system will prevent the update from being applied to the replica. The replication process may be halted to take snapshot of the replica database during the halt, and resumed after the snapshot. Accordingly, if the problematic update is later confirmed to be invalid, the replica database can be rolled back to a point before the problematic update. Advantageously, the disclosed system can be used to implement a variety of data validations on top of a database replication process to protect the replica database from receiving undesired updates.
1 . A system, comprising:
one or more hardware processors and associated memory storing program instructions executable by the one or more hardware processors to implement a database system configured to:
execute a database replication process from a first database to a second database, wherein the database replication process is configured to:
(a) delay updates of the first database from being applied to the second database for a delay period; and
(b) prevent an update of the first database from being applied to the second database responsive to a data validation failure for the update detected during the delay period;
maintain a buffer of updates performed on the first database waiting to be validated and applied to the second database;
track a buffer size metric of the buffer indicating an amount of updates in the buffer waiting to be validated;
track a validation time metric indicating an amount of time used to detect data validation failures; and
dynamically adjust the delay period based at least in part on the buffer size metric and the validation time metric.
2 . The system of claim 1 , wherein the database system is configured to:
dynamically decrease the delay period for one or more increasing values of the buffer size metric; and
dynamically increase the delay period for one or more increasing values of the validation time metric.
3 . The system of claim 2 , wherein the database system is configured to:
in conjunction with decreasing the delay period, increase a number of validator instances according to an auto-scaling mechanism.
4 . The system of claim 1 , wherein the buffer size metric comprises one or more of:
a number of updates currently in the buffer,
a number of updates in the buffer in a previous time window,
a number of updates entering or exiting the buffer,
an amount of storage space used by the buffer,
an amount of data entering or exiting the buffer,
a rate of change of a number of updates in the buffer,
a predicted number of updates in the buffer for a next time period, and
a predicted amount of storage space used by the buffer for a next time period.
5 . The system of claim 1 , wherein the validation time metric is measured for an external data validator that is remote from the database system.
6 . The system of claim 5 , wherein the external data validator is a blockchain validator in a blockchain system.
7 . The system of claim 1 , wherein the validation time metric comprises one or more of:
an average time to receive validations in a last time period,
a trend or rate of change in validation times, and
a measure of a validating performing data validation checks.
8 . The system of claim 1 , wherein the dynamic adjustments of the delay period is controlled by one or more user-configured policies, rules, or settings.
9 . The system of claim 1 , wherein the database system is configured to, after the data validation failure is detected:
halt the database replication process; and
promote the second database to replace the first database.
10 . The system of claim 1 , wherein the database system is configured to, after the data validation failure is detected:
create a snapshot of the second database as a checkpoint; and
resume the database replication process.
11 . A method, comprising:
executing a database system implemented by one or more hardware processors with associated memory, including:
executing a database replication process from a first database to a second database, wherein the database replication process includes:
(a) delaying updates of the first database from being applied to the second database for a delay period; and
(b) preventing an update of the first database from being applied to the second database responsive to a data validation failure for the update detected during the delay period;
maintaining a buffer of updates performed on the first database waiting to be validated and applied to the second database;
tracking a buffer size metric of the buffer indicating an amount of updates in the buffer waiting to be validated;
tracking a validation time metric indicating an amount of time used to detect data validation failures; and
dynamically adjusting the delay period based at least in part on the buffer size metric and the validation time metric.
12 . The method of claim 11 , further comprising the database system:
dynamically decreasing the delay period for one or more increasing values of the buffer size metric; and
dynamically increasing the delay period for one or more increasing values of the validation time metric.
13 . The method of claim 12 , further comprising the database system:
in conjunction with decreasing the delay period, increasing a number of validator instances according to an auto-scaling mechanism.
14 . The method of claim 11 , wherein the buffer size metric comprises one or more of:
a number of updates currently in the buffer,
a number of updates in the buffer in a previous time window,
a number of updates entering or exiting the buffer,
an amount of storage space used by the buffer,
an amount of data entering or exiting the buffer,
a rate of change of a number of updates in the buffer,
a predicted number of updates in the buffer for a next time period, and
a predicted amount of storage space used by the buffer for a next time period.
15 . The method of claim 11 , wherein the validation time metric is measured for an external data validator that is remote from the database system.
16 . The method of claim 15 , wherein the external data validator is a blockchain validator in a blockchain system.
17 . The method of claim 11 , wherein the validation time metric comprises one or more of:
an average time to receive validations in a last time period,
a trend or rate of change in validation times, and
a measure of a validating performing data validation checks.
18 . The method of claim 11 , wherein the dynamic adjustments of the delay period is controlled by one or more user-configured policies, rules, or settings.
19 . The method of claim 11 , further comprising the database system,
after detecting the data validation failure is detected:
halting the database replication process; and
promoting the second database to replace the first database.
20 . The method of claim 11 , further comprising the database system,
after detecting the data validation failure:
creating a snapshot of the second database as a checkpoint; and
resuming the database replication process.