High-performance database replication systems and methods

1. A method of performing database replication, the method comprising:

generating, at a producer cluster, a binary change log based on a write transaction made to a database partition in the producer cluster;

including with the binary change log a sequence number corresponding to the write transaction;

transmitting the binary change log from the producer cluster to a consumer cluster;

determining, at the consumer cluster, whether to apply the binary change log based on the sequence number included with the binary change log;

applying the binary change log to a database partition in the consumer cluster; and

storing, at the consumer cluster, the sequence number by combining the sequence number with a numerical range comprising a first number and a second number, the second number comprising a value corresponding to a most recent binary change log applied to the database partition in the consumer cluster, wherein combining the sequence number with the numerical range comprises:

determining that the sequence number consecutively follows the second number, and replacing the second number with the sequence number; or

determining that the sequence number does not consecutively follow the second number, and storing at the consumer cluster a second numerical range comprising the sequence number.

2. The method of claim 1 , wherein including with the binary change log a sequence number corresponding to the write transaction comprises increasing a previous sequence number associated with the database partition in the producer cluster.

3. The method of claim 1 , wherein determining whether to apply the binary change log comprises determining, based on the sequence number included with the binary change log, whether the binary change log has already been applied to the database partition in the consumer cluster.

4. The method of claim 1 , wherein the first number comprises a value corresponding to an earliest binary change log applied to the database partition in the consumer cluster.

5. The method of claim 1 , wherein the first number comprises a value indicating that all binary change logs, from an earliest binary change log created for the database partition in the producer cluster to the most recent binary change log, have been applied to the database partition in the consumer cluster.

6. The method of claim 1 , further comprising:

receiving, for each database partition in a plurality of database partitions in the consumer cluster, a numerical range of sequence numbers representing binary change logs applied to the respective database partition;

determining, based on the received numerical ranges, which binary change logs were applied to the consumer cluster; and

sending a request to the producer cluster, by the consumer cluster, for one or more binary change logs that were not applied to one or more database partitions in the consumer cluster.

7. The method of claim 1 , wherein the producer cluster comprises a first topology and the consumer cluster comprises a second, different topology, the method further comprising:

receiving, at the consumer cluster, a plurality of binary change logs associated with a first database partition in the producer cluster, each of the binary change logs including a different sequence number;

applying the plurality of binary change logs to a plurality of database partitions in the consumer cluster; and

for each applied binary change log, storing, at the consumer cluster, the sequence number for that binary change log in a respective numerical range associated with the database partition in the consumer cluster to which that binary change log was applied.

8. The method of claim 7 , further comprising:

receiving the numerical ranges associated with the database partitions in the consumer cluster;

merging the numerical ranges to form at least one merged numerical range representing the plurality of binary change logs applied to the database partitions; and

identifying, based on the at least one merged numerical range, which sequence numbers and corresponding binary logs are missing with respect to the first database partition in the producer cluster.

9. A system for performing database replication, the system comprising:

at least one memory for storing computer-executable instructions; and

at least one processor for executing the instructions stored on the at least one memory, wherein execution of the instructions programs the at least one processor to perform operations comprising:

generating, at a producer cluster, a binary change log based on a write transaction made to a database partition in the producer cluster;

including with the binary change log a sequence number corresponding to the write transaction;

transmitting the binary change log from the producer cluster to a consumer cluster;

determining, at the consumer cluster, whether to apply the binary change log based on the sequence number included with the binary change log;

applying the binary change log to a database partition in the consumer cluster; and

storing, at the consumer cluster, the sequence number by combining the sequence number with a numerical range comprising a first number and a second number, the second number comprising a value corresponding to a most recent binary change log applied to the database partition in the consumer cluster, wherein combining the sequence number with the numerical range comprises:

determining that the sequence number consecutively follows the second number, and replacing the second number with the sequence number; or

determining that the sequence number does not consecutively follow the second number, and storing at the consumer cluster a second numerical range comprising the sequence number.

10. The system of claim 9 , wherein including with the binary change log a sequence number corresponding to the write transaction comprises increasing a previous sequence number associated with the database partition in the producer cluster.

11. The system of claim 9 , wherein determining whether to apply the binary change log comprises determining, based on the sequence number included with the binary change log, whether the binary change log has already been applied to the database partition in the consumer cluster.

12. The system of claim 9 , wherein the first number comprises a value corresponding to an earliest binary change log applied to the database partition in the consumer cluster.

13. The system of claim 9 , wherein the first number comprises a value indicating that all binary change logs, from an earliest binary change log created for the database partition in the producer cluster to the most recent binary change log, have been applied to the database partition in the consumer cluster.

14. The system of claim 9 , wherein the operations further comprise:

receiving, for each database partition in a plurality of database partitions in the consumer cluster, a numerical range of sequence numbers representing binary change logs applied to the respective database partition;

determining, based on the received numerical ranges, which binary change logs were applied to the consumer cluster; and

sending a request to the producer cluster, by the consumer cluster, for one or more binary change logs that were not applied to one or more database partitions in the consumer cluster.

15. The system of claim 9 , wherein the producer cluster comprises a first topology and the consumer cluster comprises a second, different topology, and wherein the operations further comprise:

receiving, at the consumer cluster, a plurality of binary change logs associated with a first database partition in the producer cluster, each of the binary change logs including a different sequence number;

applying the plurality of binary change logs to a plurality of database partitions in the consumer cluster; and

for each applied binary change log, storing, at the consumer cluster, the sequence number for that binary change log in a respective numerical range associated with the database partition in the consumer cluster to which that binary change log was applied.

16. The system of claim 15 , wherein the operations further comprise:

receiving the numerical ranges associated with the database partitions in the consumer cluster;

merging the numerical ranges to form at least one merged numerical range representing the plurality of binary change logs applied to the database partitions; and

identifying, based on the at least one merged numerical range, which sequence numbers and corresponding binary logs are missing with respect to the first database partition in the producer cluster.