Multi-instance redo apply
A method, apparatus, and system for multi-instance redo apply is provided for standby databases. A multi-instance primary database generates a plurality of redo records, which are received and applied by a physical standby running a multi-instance standby database. Each standby instance runs a set of processes that utilize non-blocking, single-task threads for high parallelism. At each standby instance for the multi-instance redo, the plurality of redo records are merged into a stream from one or more redo strands in logical time order, distributed to standby instances according to determined apply slave processes using an intelligent workload distribution function, reemerged after receiving updates from remote instances, and applied in logical time order by the apply slave processes. Redo apply progress is tracked at each instance locally and also globally, allowing a consistent query logical time to be maintained and published to service database read query requests concurrently with the redo apply.
1. A method comprising:
on each standby instance of a plurality of standby instances for a standby database:
an apply process of a plurality of apply processes running on said each standby instance to apply change records to said standby database;
a sender process of a plurality of sender processes running on said each standby instance to transmit change records to at least one receiver process of a plurality of receiver processes;
a receiver process of said plurality of receiver processes running on said each standby instance to receive change records transmitted by at least one sender process of said plurality of sender processes to said receiver process;
each standby instance of said plurality of standby instances:
storing a plurality of change records received from one or more primary instances for a primary multi-instance database;
making determinations of how to assign said plurality of change records to said plurality of apply processes, each determination of said determinations assigning each change record of said plurality of change records to an apply process of said plurality of apply processes;
according to said determinations, the respective sender process of said plurality of sender processes running on said each standby instance dispatching each change record of at least of a portion of said plurality of change records to a standby instance on which the apply process assigned to said each change record runs;
each receiver process of said plurality of receiver processes receiving dispatched change records dispatched to said each receiver process by a sender process running on a standby instance different than the standby instance on which said receiver process runs;
each standby instance of said plurality of standby instances ordering a respective plurality of assigned change records assigned to an apply process running on said each standby instance, said ordering made according to a logical time associated with each assigned change record of said plurality of assigned change records, said respective plurality of assigned change records including dispatched change records received by a receiver process of said plurality of receiver processes running on said each standby instance transmitted by a sender process of said plurality of sender processes running on a different standby instance of said plurality of standby instances;
each standby instance of said plurality of standby instances distributing each assigned change record of said respective plurality of assigned change records to the assigned apply process of said each assigned change record.
2. The method of claim 1 , further comprising:
for each standby instance of said plurality of standby instances:
maintaining a local influx logical time such that said respective plurality of
assigned change records are applied by said assigned apply process for each said assigned change record up to said local influx logical time;
for a particular standby instance of said plurality of standby instances:
maintaining a global influx logical time corresponding to a smallest of said local influx logical time of said plurality of standby instances.
3. The method of claim 2 , further comprising:
for a particular standby instance of said plurality of standby instances:
providing a response to a read request for said standby database that is consistent to a query logical time that is no greater than said global influx logical time.
4. The method of claim 2 , further comprising:
ascertaining a crashed standby instance of said plurality of standby instances, a crash logical time, and a clean logical time corresponding to said global influx logical time;
restarting said determinations for a recovery subset of said plurality of change records including change records having logical times from said crash logical time until said clean logical time;
resuming said determinations for said plurality of change records.
5. The method of claim 1 , wherein each of said plurality of change records is associated with a particular database block address (DBA), and wherein for each standby instance of said plurality of standby instances:
each determination of said determinations distributes each change record of said plurality of change records to said assigned apply process of said apply processes based on a distribution function on said DBA associated with said each change record.
6. The method of claim 5 , wherein said distribution function is based on a hash function.
7. The method of claim 5 , wherein said distribution function further distributes each said plurality of change records according to said DBA having an affinity to a standby instance on which said assigned apply process runs.
8. The method of claim 7 , wherein said affinity is to an undo tablespace on said standby instance on which said assigned apply process runs.
9. The method of claim 5 , wherein said distribution function further distributes each said plurality of change records to a local standby instance for direct loads.
10. The method of claim 5 , wherein said distribution function further distributes each said plurality of change records such that each said plurality of standby instances receive change records within a contiguous range of DBAs.
11. The method of claim 1 , further comprising:
for a particular standby instance of said plurality of standby instances:
reading a metadata marker from said plurality of change records;
halting said dispatching until said plurality of apply processes applies all change records from said plurality of change records having a logical time prior to a logical time of said metadata marker.
12. The method of claim 1 , further comprising:
for a particular standby instance of said plurality of standby instances:
reading a metadata marker from said plurality of change records;
forwarding said metadata marker to be sorted and applied by a coordinator process of a master coordinator instance of said plurality of standby instances.
13. The method of claim 12 , wherein said metadata marker is a library cache invalidation marker, and wherein the coordinator process instructs each of the plurality of standby instances to purge a respective library cache.
14. The method of claim 1 , wherein said logical time is a system commit number (SCN).
15. A standby database comprising one or more computing devices, the standby database including:
a plurality of standby instances each configured to run:
an apply process of a plurality of apply processes to apply change records to said standby database;
a sender process of a plurality of sender processes to transmit change records to at least one receiver process of a plurality of receiver processes;
a receiver process of said plurality of receiver processes to receive change records transmitted by at least one sender process of said plurality of sender processes to said receiver process;
each standby instance of said plurality of standby instances further configured to:
store a plurality of change records received from one or more primary instances for a primary multi-instance database;
make determinations of how to assign said plurality of change records to said plurality of apply processes, each determination of said determinations assigning each change record of said plurality of change records to an apply process of said plurality of apply processes;
according to said determinations, the respective sender process of said plurality of sender processes running on said each standby instance dispatching each change record of at least of a portion of said plurality of change records to a standby instance on which the apply process assigned to said each change record runs;
each receiver process of said plurality of receiver processes configured to receive dispatched change records dispatched to said each receiver process by a sender process running on a standby instance different than the standby instance on which said receiver process runs;
each standby instance of said plurality of standby instances further configured to order a respective plurality of assigned change records assigned to an apply process running on said each standby instance, said ordering made according to a logical time associated with each assigned change record of said plurality of assigned change records, said respective plurality of assigned change records including dispatched change records received by a receiver process of said plurality of receiver processes running on said each standby instance transmitted by a sender process of said plurality of sender processes running on a different standby instance of said plurality of standby instances;
each standby instance of said plurality of standby instances further configured to distribute each assigned change record of said respective plurality of assigned change records to the assigned apply process of said each assigned change record.
16. The standby database of claim 15 , wherein:
each standby instance of said plurality of standby instances is further configured to:
maintain a local influx logical time such that said respective plurality of assigned change records are applied by said assigned apply process for each said assigned change record up to said local influx logical time;
a particular standby instance of said plurality of standby instances is configured to:
maintain a global influx logical time corresponding to a smallest of said local influx logical time of said plurality of standby instances;
provide a response to a read request for said standby database that is consistent to a query logical time that is no greater than said global influx logical time.
17. The standby database of claim 15 , wherein each of said plurality of change records is associated with a particular database block address (DBA), and wherein for each standby instance of said plurality of standby instances:
each determination of said determinations is configured to distribute each change record of said plurality of change records to said assigned apply process of said apply processes based on a distribution function on said DBA associated with said each change record, wherein said distribution function further distributes each said plurality of change records according to said DBA having an affinity to a standby instance on which said assigned apply process runs.
18. One or more non-transitory computer-readable storage media, storing instructions which, when executed by one or more processors, cause:
for each standby instance of a plurality of standby instances for a standby database:
running an apply process of a plurality of apply processes on said each standby instance to apply change records to said standby database;
running a sender process of a plurality of sender processes on said each standby instance to transmit change records to at least one receiver process of a plurality of receiver processes;
running a receiver process of said plurality of receiver processes on said each
standby instance to receive change records transmitted by at least one sender process of said plurality of sender processes to said receiver process;
each standby instance of said plurality of standby instances:
storing a plurality of change records received from one or more primary instances for a primary multi-instance database;
making determinations of how to assign said plurality of change records to said plurality of apply processes, each determination of said determinations assigning each change record of said plurality of change records to an apply process of said plurality of apply processes;
according to said determinations, the respective sender process of said plurality of sender processes running on said each standby instance dispatching each change record of at least of a portion of said plurality of change records to a standby instance on which the apply process assigned to said each change record runs;
each receiver process of said plurality of receiver processes receiving dispatched change records dispatched to said each receiver process by a sender process running on a standby instance different than the standby instance on which said receiver process runs;
each standby instance of said plurality of standby instances ordering a respective plurality of assigned change records assigned to an apply process running on said each standby instance, said ordering made according to a logical time associated with each assigned change record of said plurality of assigned change records, said respective plurality of assigned change records including dispatched change records received by a receiver process of said plurality of receiver processes running on said each standby instance transmitted by a sender process of said plurality of sender processes running on a different standby instance of said plurality of standby instances;
each standby instance of said plurality of standby instances distributing each assigned change record of said respective plurality of assigned change records to the assigned apply process of said each assigned change record.
19. The one or more non-transitory computer-readable storage media of claim 18 , storing instructions which, when executed by the one or more processors, further cause:
for each standby instance of said plurality of standby instances:
maintaining a local influx logical time such that said respective plurality of assigned change records are applied by said assigned apply process for each said assigned change record up to said local influx logical time;
for a particular standby instance of said plurality of standby instances:
maintaining a global influx logical time corresponding to a smallest of said local influx logical time of said plurality of standby instances;
providing a response to a read request for said standby database that is consistent to a query logical time that no greater than said global influx logical time.
20. The one or more non-transitory computer-readable storage media of claim 19 , storing instructions which, when executed by the one or more processors, further cause:
for a particular standby instance of said plurality of standby instances:
providing a response to a read request for said standby database that is consistent to a query logical time that is no greater than said global influx logical time.
21. The one or more non-transitory computer-readable storage media of claim 19 , storing instructions which, when executed by the one or more processors, further cause:
ascertaining a crashed standby instance of said plurality of standby instances, a crash logical time, and a clean logical time corresponding to said global influx logical time;
restarting said determinations for a recovery subset of said plurality of change records including change records having logical times from said crash logical time until said clean logical time;
resuming said determinations for said plurality of change records.
22. The one or more non-transitory computer-readable storage media of claim 18 , wherein each of said plurality of change records is associated with a particular database block address (DBA), and wherein for each standby instance of said plurality of standby instances:
each determination of said determinations distributes each change record of said plurality of change records to said assigned apply process of said apply processes based on a distribution function on said DBA associated with said each change record, wherein said distribution function further distributes each said plurality of change records according to said DBA having an affinity to a standby instance on which said assigned apply process runs.
23. The one or more non-transitory computer-readable storage media of claim 22 , wherein said distribution function is based on a hash function.
24. The one or more non-transitory computer-readable storage media of claim 22 , wherein said affinity is to an undo tablespace on said standby instance on which said assigned apply process runs.
25. The one or more non-transitory computer-readable storage media of claim 18 , wherein each of said plurality of change records is associated with a particular database block address (DBA), and wherein for each standby instance of said plurality of standby instances:
each determination of said determinations distributes each change record of said plurality of change records to said assigned apply process of said apply processes based on a distribution function on said DBA associated with said each change record.
26. The one or more non-transitory computer-readable storage media of claim 25 , wherein said distribution function further distributes each said plurality of change records to a local standby instance for direct loads.
27. The one or more non-transitory computer-readable storage media of claim 25 , wherein said distribution function further distributes each said plurality of change records such that each said plurality of standby instances receive change records within a contiguous range of DBAs.
28. The one or more non-transitory computer-readable storage media of claim 18 , storing instructions which, when executed by the one or more processors, further cause:
for a particular standby instance of said plurality of standby instances:
reading a metadata marker from said plurality of change records;
halting said dispatching until said plurality of apply processes applies all change records from said plurality of change records having a logical time prior to a logical time of said metadata marker.
29. The one or more non-transitory computer-readable storage media of claim 18 , storing instructions which, when executed by the one or more processors, further cause:
for a particular standby instance of said plurality of standby instances:
reading a metadata marker from said plurality of change records;
forwarding said metadata marker to be sorted and applied by a coordinator process of a master coordinator instance of said plurality of standby instances.
30. The one or more non-transitory computer-readable storage media of claim 29 , wherein said metadata marker is a library cache invalidation marker, and wherein the coordinator process instructs each of the plurality of standby instances to purge a respective library cache.
31. The one or more non-transitory computer-readable storage media of claim 18 , wherein said logical time is a system commit number (SCN).