System and method of providing system availability by preventing software crash due to recovery failure
Disclosed is an improved approach to handle recovery failures associated with fatal recovery processes. A software crash is prevented when a program has a failure that would typically crash the software, involving analyzing the failure in an operation, accumulating the failure information, clearing the failure (hence preventing the immediate crash), and not performing the desired operation until the root cause of the failure is fixed.
1 . A computer implemented method, comprising:
identifying a transaction that operates against a database instance and is associated with a failure situation;
responsive to identifying the transaction that has failed, performing transaction recovery upon the transaction with a recovery process;
during the transaction recovery, encountering a recovery failure during the transaction recovery; and
responsive to both identifying the transaction that has failed and encountering the recovery failure, quarantining the transaction from the transaction recovery at least by isolating one or more transaction operations from the transaction recovery based at least in part upon the recovery failure being encountered for the transaction.
2 . The method of claim 1 , wherein quarantining the transaction causes the recovery process to skip the transaction when performing the transaction recovery so that the transaction recovery is performed on one or more other transactions instead of the transaction that is quarantined, and that an uncommitted state of the transaction is prevented from being corrected in the database instance.
3 . The method of claim 1 , wherein a quarantine limit is configured, and exceeding the quarantine limit causes an escalation of quarantine to encompass a scope greater than an individual transaction.
4 . The method of claim 3 , wherein the scope of the quarantine is escalated to an entire database or the database instance.
5 . The method of claim 3 , wherein the transaction is performed in a multi-tenant database having multiple private databases, and the scope of the quarantine is escalated to one or more private databases of the multiple private databases.
6 . The method of claim 1 , wherein a second transaction performs one or more database operations on a block affected by the transaction that is quarantined, where the one or more database operations by the second transaction on the block are not prevented since block level recovery is applied to the block.
7 . The method of claim 1 , wherein quarantine resolution is applied before releasing the transaction from quarantine, where the quarantine resolution comprises at least one of block media recovery, segment repair, or point in time recovery, quarantining the transaction prevents an uncommitted state of the transaction from being corrected in the database instance, and the failure situation is caused by an operation of a software program for executing the transaction.
8 . A tangible computer program product embodied on a computer usable medium, the tangible computer readable medium having stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute actions comprising:
identifying a transaction that operates against a database instance and is associated with a failure situation;
responsive to identifying the transaction that has failed, performing transaction recovery upon the transaction with a recovery process;
during the transaction recovery, encountering a recovery failure during the transaction recovery; and
responsive to both identifying the transaction that has failed and encountering the recovery failure, quarantining the transaction from the transaction recovery at least by isolating one or more transaction operations from the transaction recovery based at least in part upon the recovery failure being encountered for the transaction.
9 . The computer program product of claim 8 , wherein quarantining the transaction causes the recovery process to skip the transaction when performing the transaction recovery so that the transaction recovery is performed on one or more other transactions instead of the transaction that is quarantined, and that an uncommitted state of the transaction is prevented from being corrected in the database instance.
10 . The computer program product of claim 8 , the actions further comprise configuring a quarantine limit, and exceeding the quarantine limit causes an escalation of quarantine to encompass a scope greater than an individual transaction.
11 . The computer program product of claim 10 , wherein the actions further comprise escalating the scope of the quarantine to an entire database or the database instance.
12 . The computer program product of claim 10 , wherein the transaction is performed in a multi-tenant database having multiple private databases, and the actions further comprise escalating the scope of the quarantine to one or more private databases of the multiple private databases.
13 . The computer program product of claim 8 , wherein a second transaction performs one or more database operations on a block affected by the transaction that is quarantined, where the one or more database operations by the second transaction on the block are not prevented since block level recovery is applied to the block.
14 . The computer program product of claim 8 , wherein quarantine resolution is applied before releasing the transaction from quarantine, where the quarantine resolution comprises at least one of block media recovery, segment repair, or point in time recovery, quarantining the transaction prevents an uncommitted state of the transaction from being corrected in the database instance, and the failure situation is caused by an operation of a software program for executing the transaction.
15 . A computer-based system, comprising:
a computer processor to execute a set of program code instructions;
a memory to hold the program code instructions, in which the program code instructions comprises program code which, when executed by the computer processor, causes the computer processor to perform a set of acts, the set of acts comprising:
identifying a transaction that operates against a database instance and is associated with a failure situation;
responsive to identifying the transaction that has failed, performing transaction recovery upon the transaction with a recovery process;
during the transaction recovery, encountering a recovery failure during the transaction recovery; and
responsive to both identifying the transaction that has failed and encountering the recovery failure, quarantining the transaction from the transaction recovery at least by isolating one or more transaction operations from the transaction recovery based at least in part upon the recovery failure being encountered for the transaction.
16 . The system of claim 15 , wherein quarantining the transaction causes the recovery process to skip the transaction when performing the transaction recovery so that the transaction recovery is performed on one or more other transactions instead of the transaction that is quarantined, and that an uncommitted state of the transaction is prevented from being corrected in the database instance.
17 . The system of claim 15 , wherein a quarantine limit is configured, and exceeding the quarantine limit causes an escalation of quarantine to encompass a scope greater than an individual transaction.
18 . The system of claim 17 , wherein the scope of the quarantine is escalated to an entire database or the database instance.
19 . The system of claim 17 , wherein the transaction is performed in a multi-tenant database having multiple private databases, and the scope of the quarantine is escalated to one or more private databases of the multiple private databases.
20 . The system of claim 15 , wherein a second transaction performs one or more database operations on a block affected by the transaction that is quarantined, where the or more database operations by the second transaction on the block are not prevented since block level recovery is applied to the block.
21 . The system of claim 15 , wherein quarantine resolution is applied before releasing the transaction from quarantine, where the quarantine resolution comprises at least one of block media recovery, segment repair, or point in time recovery, quarantining the transaction prevents an uncommitted state of the transaction from being corrected in the database instance, and the failure situation is caused by an operation of a software program for executing the transaction.