Parallel truncation point management for log based replication
View Patent ↗Disclosed herein are system, method, and computer program product embodiments for replicating data in a distributed database system. Data containing a replicated truncation point associated with a replicating system is received via a data path. It can then be determined that the truncation point represents the point at which all data in a transaction log has been replicated (e.g., successfully or safely) and the transaction log can then be truncated at the truncation point (i.e., the data up to the truncation point deflected). Data containing an additional replicated truncation point associated with an additional replicating system via an additional data path may be received. It can then be determined that the additional replicated truncation point represents the point at which all data in the transaction log has been replicated and the transaction log can be then truncated at the additional replicated truncation point.
1. A method, comprising:
sending, by a plurality of sender threads, a request for truncation points to a plurality of replicating systems, wherein each of the plurality of replicating systems is associated with a corresponding sender thread of the plurality of sender threads;
receiving, by the plurality of sender threads and in response to the sent request, data containing a plurality of replicated truncation points associated with the plurality of replicating systems via a plurality of data paths;
selecting a safe point from among the plurality of received replicated truncation points, the safe point representing a point at which all data in a transaction log has been replicated by the plurality of replicating systems; and
truncating the transaction log at the determined safe point.
2. The method of claim 1 , comprising:
receiving data containing an additional replicated truncation point associated with an additional replicating system via an additional one of the plurality of data paths;
determining that the additional replicated truncation point represents the point at which all data in the transaction log has been replicated; and
truncating the transaction log at the additional replicated truncation point.
3. The method of claim 1 , further comprising:
sending data to a replicating system, wherein the data comprises transaction log data and an identifier indicating the point in the transaction log where the transaction log data ends.
4. The method of claim 3 , wherein the data containing the replicated truncation point is received in response to sending the data to the replicating system.
5. The method of claim 4 , wherein the replicated truncation point is configured to identify a point in the transaction log up to which all data has been replicated to the replicating system.
6. The method of claim 1 , further comprising:
sending a request to a replicating system for an updated replication truncation point.
7. The method of claim 6 , further comprising:
periodically repeating the request to the replicating system.
8. The method of claim 1 , further comprising:
storing the plurality of replicated truncation points in a memory.
9. The method of claim 1 , further comprising:
sending a command to one of the plurality replicating systems instructing it to update its associated replicated truncation point.
10. The method of claim 1 , wherein selecting the safe point from among the plurality of received replicated truncation points comprises selecting an oldest of the plurality of received replicated truncation points.
11. The method of claim 1 , wherein each of the plurality of sender threads is associated with one of a plurality of data paths.
12. The method of claim 11 , wherein each of the plurality of replicating systems is associated with one of the plurality of data paths.
13. A system, comprising:
a communication interface communicatively coupled to a plurality of data paths; and
one or more processors communicatively coupled to the communication interface and configured to:
send, using a plurality of sender threads, a request for truncation points to a plurality of replicating systems, wherein each of the plurality of replicating systems is associated with a corresponding sender thread of the plurality of sender threads, and
receive, in response to the sent request, data containing a plurality of replicated truncation points associated with the plurality of replicating systems via the plurality of data paths;
select a safe point from among the plurality of received replicated truncation points, the safe point representing a point at which all data in a transaction log has been replicated by the plurality of replicating systems; and
truncate the transaction log at the determined safe point.
14. The system of claim 13 , wherein the one or more processors are further configured to receive data containing an additional replicated truncation point associated with an additional replicating system via one of the plurality of data paths.
15. The system of claim 14 , wherein the one or more processors are further configured to:
determine that the additional replicated truncation point represents the point at which all data in the transaction log has been replicated; and
truncate the transaction log at the additional replicated truncation point.
16. The system of claim 13 , wherein the one or more processors are further configured to send, via the communication interface, data to the replicating system, wherein the data comprises transaction log data and an identifier indicating the point in the transaction log where the transaction log data ends.
17. The system of claim 13 , wherein the one or more processors are further configured to identify a point in the transaction log up to which all data has been replicated to the replicating system based on the replicated truncation point.
18. The system of claim 13 , wherein the one or more processors are further configured to determine that one of the plurality of received replicated truncation points represents the safe point by determining that the received truncation point is the oldest truncation point.
19. The system of claim 13 , wherein the communication interface is further configured to send a request to a replicating system for an updated replication truncation point.
20. The system of claim 19 , wherein the communication interface is configured to periodically repeat the request to the replicating system.
21. The system of claim 13 , wherein the communication interface is further configured to send a command to one of the plurality of replicating systems instructing it to update its associated replicated truncation point.
22. A non-transitory computer readable medium having computer instructions stored thereon that, when executed by one or more processors, cause those processors to perform operations, comprising:
sending, by a plurality of sender threads, a request for truncation points to a plurality of replicating systems, wherein each of the plurality of replicating systems is associated with a corresponding sender thread of the plurality of sender threads;
receiving, by the plurality of sender threads and in response to the sent request, data containing a plurality of replicated truncation points associated with the plurality of replicating systems via a plurality of data paths;
selecting a safe point from among the plurality of received replicated truncation points, the safe point representing a point at which all data in a transaction log has been replicated by the plurality of replicating systems; and
truncating the transaction log at the determined safe point.