Granted Patent B2

US 12,259,905 · App. 17/448,713 · Granted Mar 25, 2025

Inventors: Felix Beier (Haigerloch, DE); Dennis Butterstein (Stuttgart, DE); Einar Lueck (Filderstadt, DE); Sabine Perathoner-Tschaffler (Nufringen, DE)

Assignee: International Business Machines Corporation

G06F16/27G06F11/1471G06F16/2255G06F16/2358

Abstract

Claims (40)

1. A computer implemented method for data synchronization in a data analysis system, the data analysis system comprising a target database system and a source database system, the method comprising

retrieving a change record describing an operation performed on a data record in the source database system of the data analysis system based on reading a transaction log associated with the source database system with a frequency higher than a defined minimum frequency;

determining a distribution key that is configured to be used by the target database system to distribute records over target database nodes of the target database system, wherein the target database system comprises a metadata catalog that includes cluster metadata and table metadata, the table meta comprising information on a total number of the target database nodes and storage properties of the target database nodes and the cluster metadata comprising information on the distribution key;

reading the change record for determining a value of the distribution key of the data record;

using the value of the distribution key for selecting a target database node of the target database nodes where the operation is to be performed, wherein selecting the target database node for storing data record comprises providing a distribution map of hash values to connection numbers, wherein each connection number indicates a connection between the source database system and a respective target database node, computing a hash value of the determined value of the distribution key of the data record, and using the distribution map for assigning the computed hash value to a connection number, wherein the connection is established according to the connection number;

establishing a direct TCP/IP connection between the source database system and the selected target database node; and

providing the change record to the selected target database node through the direct connection.

2. The method of claim 1 , repeating the method for further received change records, thereby distributing the change records to respective target database nodes through respective direct connections.

3. The method of claim 2 , the method being concurrently performed for the change records.

4. The method of claim 1 , further comprising

determining a distribution rule of the target database system, the distribution rule assigning values of the distribution key to respective target database nodes;

wherein selecting the target database node comprises applying by the source database system the distribution rule on the determined value of the distribution key.

5. The method of claim 1 , wherein receiving the change record comprises reading a transaction recovery log indicating transactions to be replicated to the target database system.

6. The method of claim 1 , wherein the operation includes at least one of inserting, deleting or updating a data record.

7. The method of claim 1 , the distribution key comprising one or more attributes of the data record.

8. A computer program product for data synchronization in a data analysis system, the data analysis system comprising a target database system and a source database system, the computer program product comprising a non-transitory computer readable hardware storage device, and program instructions stored on the computer readable hardware storage device, to:

retrieve a change record describing an operation performed on a data record in the source database system of the data analysis system based on reading a transaction log associated with the source database system with a frequency higher than a defined minimum frequency;

determine a distribution key that is configured to be used by the target database system to distribute records over target database nodes of the target database system, wherein the target database system comprises a metadata catalog that includes cluster metadata and table metadata, the table meta comprising information on a total number of the target database nodes and storage properties of the target database nodes and the cluster metadata comprising information on the distribution key;

read the change record for determining a value of the distribution key of the data record;

use the value of the distribution key for selecting a target database node of the target database nodes where the operation is to be performed, wherein selecting the target database node for storing data record comprises providing a distribution map of hash values to connection numbers, wherein each connection number indicates a connection between the source database system and a respective target database node, computing a hash value of the determined value of the distribution key of the data record, and using the distribution map for assigning the computed hash value to a connection number, wherein the connection is established according to the connection number;

establish a direct TCP/IP connection between the source database system and the selected target database node; and

provide the change record to the selected target database node through the direct connection.

9. The computer program product of claim 8 , wherein the computer readable storage device further comprises instructions to repeatedly receive change records, thereby distributing the change records to respective target database nodes through respective direct connections.

10. The computer program product of claim 9 , wherein the computer readable storage device further comprises instructions that are concurrently performed for the change records.

11. The computer program product of claim 8 , wherein the computer readable storage device further comprising instructions to:

determine a distribution rule of the target database system, the distribution rule assigning values of the distribution key to respective target database nodes; and

wherein selecting the target database node comprises applying by the source database system the distribution rule on the determined value of the distribution key.

12. The computer program product of claim 8 , wherein the computer readable storage device instructions to receive the change record further comprise reading a transaction recovery log indicating transactions to be replicated to the target database system.

13. The computer program product of claim 8 , wherein the operation includes at least one of inserting, deleting or updating a data record.

14. The computer program product of claim 8 , wherein the distribution key comprising one or more attributes of the data record.

15. A computer system for a data analysis system, the data analysis system comprising a source database system and a target database system, the computer system including one or more non-transitory computer-readable storage media configured to store program instructions and one or more computer processors configured to execute said program instructions store on the one or more non-transitory computer-readable storage media, the computer system being configured for:

receiving a change record describing an operation performed on a data record in the source database system;

determining a distribution key that is configured to be used by the target database system to distribute records over target database nodes of the target database system, wherein the target database system comprises a metadata catalog that includes cluster metadata and table metadata, the table meta comprising information on a total number of the target database nodes and storage properties of the target database nodes and the cluster metadata comprising information on the distribution key;

reading the change record for determining a value of the distribution key of the data record;

using the value of the distribution key for selecting a target database node of the target database nodes where the operation is to be performed, wherein selecting the target database node for storing data record comprises providing a distribution map of hash values to connection numbers, wherein each connection number indicates a connection between the source database system and a respective target database node, computing a hash value of the determined value of the distribution key of the data record, and using the distribution map for assigning the computed hash value to a connection number, wherein the connection is established according to the connection number;

establishing a direct connection to the selected target database node; and

providing the change record to the selected target database node through the direct connection.

16. The computer system of claim 15 , wherein the computer system is further configured to repeatedly receive change records, thereby distributing the change records to respective target database nodes through respective direct connections.

17. The computer system of claim 15 , wherein the computer system is comprised in the source database system.

18. The computer system of claim 15 , wherein the computer system is remotely connected to the source database system.

Assignments (1)

Continuity (1)

References Cited (27)