Replicating unlogged data changes in a database management system
Examples described herein provide systems and methods for replicating unlogged data changes in a database management system are disclosed. The method involves identifying a load operation in a transaction log, determining a log sequence number (LSN) associated with the load operation, and identifying data pages in a source database with headers containing the load LSN. Data from these pages is extracted and saved to an external file. The target database is updated by loading the extracted data from the external file, allowing for incremental replication. This approach optimizes replication efficiency by avoiding a full refresh of the target table, reducing resource consumption, and enhancing system performance.
1 . A computer-implemented method for replicating unlogged data changes in a database management system, the method comprising:
identifying a load operation in a transaction log for the database management system;
identifying a log sequence number (LSN) associated with the load operation;
identifying one or more data pages of a source database that have a header that includes a load LSN field that is written once when the data page is created for the load operation and thereafter remains unchanged which includes the LSN associated with the load operation, the header further including a page LSN field that is updated each time data stored on the page is modified;
extracting data from the one or more data pages and saving the extracted data to an external file for provision to a target database system via a communications network; and
updating a target database of the target database system by loading the extracted data in the external file into the target database
wherein, immediately upon completion of the load operation and without receiving a request from the target database system, the source database system extracts the data from the identified data pages and transmits the external file to the target database system.
2 . The computer-implemented method of claim 1 , wherein identifying the LSN associated with the load operation comprises extracting the LSN from the transaction log.
3 . The computer-implemented method of claim 1 , wherein updating the target database comprises incrementally loading the extracted data in the external file into the target database.
4 . The computer-implemented method of claim 1 , wherein the identifying the one or more data pages of the source database, the extracting data from the one or more data pages, and the saving the extracted data to the external file are performed automatically based on a completion of the load operation by the source database.
5 . The computer-implemented method of claim 1 , wherein the identifying the one or more data pages of the source database, the extracting data from the one or more data pages, and the saving the extracted data to the external file are performed based on the load operation being identified in the transaction log by a target database system of the database management system.
6 . The method of claim 1 , wherein the identifying of the one or more data pages comprises searching the source database for data pages whose headers contain a load LSN value that matches the log sequence number associated with the load operation, and combining the identified data pages into the external file prior to transfer to the target database system.
7 . The method of claim 1 , further comprising periodically transmitting the transaction log from the source database system to the target database system via a communications network, and wherein the identifying of the load operation occurs upon processing the periodically transmitted transaction log.
8 . A system comprising:
a memory comprising computer readable instructions; and
a processing device for executing the computer readable instructions, the computer readable instructions controlling the processing device to perform operations comprising:
identifying a load operation in a transaction log for a database management system;
identifying a log sequence number (LSN) associated with the load operation;
identifying one or more data pages of a source database that have a header that includes a load LSN field that is written once when the data page is created for the load operation and thereafter remains unchanged which includes the LSN associated with the load operation, the header further including a page LSN field that is updated each time data stored on the page is modified;
extracting data from the one or more data pages and saving the extracted data to an external file for provision to a target database system via a communications network; and
updating a target database of the target database system by loading the extracted data in the external file into the target database,
wherein the processing device is further configured, immediately upon completion of the load operation and without receiving a request from the target database system, to automatically extract the data from the identified data pages and transmit the external file to the target database system.
9 . The system of claim 8 , wherein identifying the LSN associated with the load operation comprises extracting the LSN from the transaction log.
10 . The system of claim 8 , wherein updating the target database comprises incrementally loading the extracted data in the external file into the target database.
11 . The system of claim 8 , wherein the identifying the one or more data pages of the source database, the extracting data from the one or more data pages, and the saving the extracted data to the external file are performed automatically based on a completion of the load operation by the source database.
12 . The system of claim 8 , wherein the identifying the one or more data pages of the source database, the extracting data from the one or more data pages, and the saving the extracted data to the external file are performed based on the load operation being identified in the transaction log by a target database system of the database management system.
13 . The system of claim 8 , wherein the processing device executes computer readable instructions that cause a database scraper module to search the source database for data pages whose headers contain a load LSN value matching the log sequence number associated with the load operation and to combine the identified data pages into an external file for provision to the target database system.
14 . The system of claim 8 , further comprising a communications network interface configured to periodically transmit the transaction log from the source database system to the target database system, wherein the processing device is configured to identify the load operation upon processing the periodically transmitted transaction log.
15 . A computer program product for replicating unlogged data changes in a database management system, the computer program product comprising:
a set of one or more computer-readable storage media;
program instructions, collectively stored in the set of one or more storage media, for causing a processor set to perform the following computer operations:
identifying a load operation in a transaction log for the database management system;
identifying a log sequence number (LSN) associated with the load operation;
identifying one or more data pages of a source database that have a header that includes a load LSN field that is written once when the data page is created for the load operation and thereafter remains unchanged which includes the LSN associated with the load operation, the header further including a page LSN field that is updated each time data stored on the page is modified;
extracting data from the one or more data pages and saving the extracted data to an external file for provision to a target database system via a communications network; and
updating a target database of the target database system by loading the extracted data in the external file into the target database,
wherein the program instructions further cause the processor set, immediately upon completion of the load operation and without receiving a request from the target database system, to extract the data from the identified data pages and transmit the external file to the target database system.
16 . The computer program product of claim 15 , wherein the program instructions further cause the processor set to search the source database for data pages whose headers contain a load LSN value matching the log sequence number associated with the load operation and to combine the identified data pages into an external file for transfer to the target database system.
17 . The computer program product of claim 15 , wherein the program instructions further cause the processor set to periodically transmit the transaction log from the source database system to the target database system via a communications network and to identify the load operation upon processing the periodically transmitted transaction log.