Continuous real-time masked database replication
A system performs masking of data stored in a database. The system receives change logs from a source database and masks the change logs to generate masked change logs. The system stores the masked change logs in a pre-buffer. If the system determines that the set of masked change logs stored in the pre-buffer corresponds to a complete transaction, the system determines whether any conflicts exist between masked change logs stored in the pre-buffer and masked change logs stored in a main buffer. If the system identifies conflicts, the system applies the masked change logs stored in the main buffer to a masked replica database. This causes the system to move masked change logs corresponding to a complete transaction from the pre-buffer to the main buffer.
1. A method for maintaining a masked replica database of a source database, comprising:
receiving one or more change logs, each corresponding to a respective update event made to data of the source database;
masking the one or more change logs using a predetermined masking algorithm to generate a set of masked change logs;
storing the set of masked change logs in a pre-buffer;
responsive to a determination that the set of masked change logs stored in the pre-buffer corresponds to a complete transaction:
determining whether any conflicts exist between any of the set of masked change logs stored in the pre-buffer and one or more masked change logs stored in a main buffer, wherein a conflict exists when a change log of the set of masked change logs and a change log of the one or more masked change logs stored in the main buffer are configured to operate on the same data of the masked replica database;
responsive to determining that a conflict exists between any of the set of masked change logs stored in the pre-buffer and the one or more masked change logs stored in the main buffer, applying the one or more masked change logs stored in the main buffer to the masked replica database to update the masked replica database, and flushing the main buffer;
moving the set of masked change logs corresponding to a complete transaction from the pre-buffer to the main buffer.
2. The method of claim 1 , further comprising applying the one or more masked change logs stored in the main buffer to the masked replica database and flushing the main buffer responsive to a determination that the main buffer is storing at least a threshold number of masked change logs.
3. The method of claim 1 , wherein applying the one or more masked change logs stored in the main buffer to the masked replica database and flushing the main buffer comprises:
transmitting the one or more masked change logs from the main buffer to the masked replica database; and
receiving a response from the masked replica database indicating whether the one or more masked change logs were successfully applied.
4. The method of claim 1 , wherein the one or more change logs comprise changed data capture (CDC) data.
5. The method of claim 1 , wherein each change log comprises a sequence number indicating a transaction that the change log is a part of, and further comprising determining that the set of masked change logs stored in pre-buffer corresponds to a complete transaction responsive to receiving a new change log having a different sequence number in comparison to the set of the masked change logs stored in the pre-buffer.
6. The method of claim 1 , further comprising determining that the set of masked change logs stored in pre-buffer corresponds to a complete transaction responsive to receiving a new change log explicitly indicating a transaction commit event.
7. The method of claim 1 , wherein the one or more change logs each correspond to one of an update, an insert, or a delete.
8. The method of claim 1 , further comprising provisioning at least one virtual database (VDB) from the masked replica database.
9. The method of claim 1 , wherein masking the one or more change logs using the predetermined masking algorithm to generate the one or more masked changes log comprises converting the one or more change logs to Java Database Connectivity (JDBC) statements.
10. The method of claim 1 , wherein masking the one or more change logs using a the predetermined masking algorithm to generate one or more masked change logs comprises, for a change log of the one or more change logs:
identifying a primary key associated with the change log;
masking the primary key in accordance with the predetermined masking algorithm to generate a masked primary key; and
using the masked primary key to identify data within the masked replica to be updated based on the change log.
11. A computer readable non-transitory storage medium, storing instructions that when executed by a computer processor cause the computer processor to perform steps comprising:
receiving one or more change logs, each corresponding to a respective update event made to data of the source database;
masking the one or more change logs using a predetermined masking algorithm to generate a set of masked change logs;
storing the set of masked change logs in a pre-buffer;
responsive to a determination that the set of masked change logs stored in the pre-buffer corresponds to a complete transaction:
determining whether any conflicts exist between any of the set of masked change logs stored in the pre-buffer and one or more masked change logs stored in a main buffer, wherein a conflict exists when a change log of the set of masked change logs and a change log of the one or more masked change logs stored in the main buffer are configured to operate on the same data of the masked replica database;
responsive to determining that a conflict exists between any of the set of masked change logs stored in the pre-buffer and the one or more masked change logs stored in the main buffer, applying the one or more masked change logs stored in the main buffer to the masked replica database to update the masked replica database, and flushing the main buffer;
moving the set of masked change logs corresponding to a complete transaction from the pre-buffer to the main buffer.
12. The computer readable non-transitory storage medium of claim 11 , further storing instructions for applying the one or more masked change logs stored in the main buffer to the masked replica database and flushing the main buffer responsive to a determination that the main buffer is storing at least a threshold number of masked change logs.
13. The computer readable non-transitory storage medium of claim 11 , wherein applying the one or more masked change logs stored in the main buffer to the masked replica database and flushing the main buffer comprises:
transmitting the one or more masked change logs from the main buffer to the masked replica database; and
receiving a response from the masked replica database indicating whether the one or more masked change logs were successfully applied.
14. The computer readable non-transitory storage medium of claim 11 , wherein the one or more change logs comprise changed data capture (CDC) data.
15. The computer readable non-transitory storage medium of claim 11 , wherein each change log comprises a sequence number indicating a transaction that the change log is a part of, and further storing instructions for determining that the set of masked change logs stored in the pre-buffer corresponds to a complete transaction responsive to receiving a new change log having a different sequence number in comparison to the set of the masked change logs stored in the pre-buffer.
16. The computer readable non-transitory storage medium of claim 11 , further storing instructions for determining that the set of masked change logs stored in pre-buffer corresponds to a complete transaction responsive to receiving a new change log explicitly indicating a transaction commit event.
17. The computer readable non-transitory storage medium of claim 11 , wherein the one or more change logs each correspond to one of an update, an insert, or a delete.
18. The computer readable non-transitory storage medium of claim 11 , further storing instructions for provisioning at least one virtual database (VDB) from the masked replica database.
19. The computer readable non-transitory storage medium of claim 11 , wherein masking the one or more change logs using the predetermined masking algorithm to generate the one or more masked change logs comprises converting the one or more change logs to Java Database Connectivity (JDBC) statements.
20. The computer readable non-transitory storage medium of claim 11 , wherein masking the one or more change logs using the predetermined masking algorithm to generate the one or more masked change logs comprises, for a change log of the one or more change logs:
identifying a primary key associated with the change log;
masking the primary key in accordance with the predetermined masking algorithm to generate a masked primary key;
using the masked primary key to identify data within the masked replica to be updated based on the change log.
21. A computer system comprising:
a computer processor; and
non-transitory computer-readable storage medium storing instructions for executing by the computer processor, the instructions for:
receiving one or more change logs, each corresponding to a respective update made to data of the source database;
masking the one or more change logs using a predetermined masking algorithm to generate one or more a set of masked change logs;
storing the one or more set of masked change logs in a pre-buffer configured to store a set of masked change logs;
responsive to a determination that the set of masked change logs stored in the pre-buffer corresponds to a complete transaction:
determining whether any conflicts exist between any of the set of masked change logs stored in the pre-buffer and one or more masked change logs stored in a main buffer, wherein a conflict exists when a change log of the set of masked change logs and a change log of the one or more masked change logs stored in the main buffer are configured to operate on the same data of the masked replica database;
responsive to determining that a conflict exists between any of the set of masked change logs stored in the pre-buffer and the one or more masked change logs stored in the main buffer, applying the one or more masked change logs stored in the main buffer to the masked replica database to update the masked replica database, and flushing the main buffer;
moving the set of masked change logs corresponding to a complete transaction from the pre-buffer to the main buffer.