Transaction conflict resolution in a file-system-backed database system
Embodiments of a transactional database system are described to implement transaction processing over database objects stored in a strongly consistent object storage system. When a transaction is initiated, the system makes a private copy of data objects that are used by the transaction. Reads and writes of the transaction will be performed on the private copy. When the transaction is to be committed, the system verifies that the committed state of the data objects has not changed outside the transaction, and updates metadata object(s) in the data storage system to point to the private copy as the currently committed state of the data objects. If the committed state of any data objects has changed during the transaction, the private copy is abandoned and the transaction is rolled back and/or retried.
1 . A system, comprising:
one or more processors with associated memory that implement a transactional database system, configured to:
manage transactions for a database, wherein the database is stored in a file system as plurality of data objects, wherein individual data objects are files in the file system that store table data associated with respective tables of the database and individual tables of the database are stored under respective table directories in the file system;
initiate a first transaction that accesses a plurality of data items in the database, including to create a first copy of one or more first data objects to be accessed by the first transaction, wherein updates in the first transaction are performed on the first copy;
determine that a committed state of the one or more first data objects has not changed since the initiation of the first transaction, and in response:
commit the first transaction, including to update one or more metadata objects in the file system to refer to the first copy as the committed state of the one or more first data objects; and
initiate a second transaction that accesses the database, including to create a second copy of one or more second data objects to be accessed by the second transaction;
determine that a committed state of the one or more second data objects has changed since the initiation of the second transaction, and in response:
roll back the second transaction, including to abandon the second copy of the one or more second data objects so that the second copy is not referred to as the committed state of the one or more second data objects; and
execute an asynchronous garbage collection process to remove copies of data objects that have been abandoned due to rollbacks, including the second copy of one or more second data objects.
2 . The system of claim 1 , wherein the transactional database system is configured to automatically retry a particular transaction when the particular transaction fails to commit due to a conflict.
3 . The system of claim 1 , wherein the transactional database system is configured to generate a warning to a client when a competing transaction is detected for a particular transaction submitted by the client.
4 . The system of claim 1 , wherein the second transaction is rolled back when the second transaction attempts to commit.
5 . The system of claim 1 , wherein the second transaction is rolled back when a competing transaction that caused the change to the committed state of the one or more second data objects is committed.
6 . The system of claim 1 , wherein the transactional database system performs conflict resolution among competing transactions based on a conflict resolution policy specified via a configuration interface.
7 . The system of claim 6 , wherein the conflict resolution policy specifies that a first finishing transaction in a group of competing transactions wins conflict resolution over one or more other transactions in the group.
8 . The system of claim 6 , wherein the conflict resolution policy specifies one or more rules to avoid starvation of transactions, wherein a particular transaction experiences starvation if it loses conflict resolution for a specified number of times.
9 . The system of claim 6 , wherein the conflict resolution policy selects a winning transaction from a group of competing transactions based on one or more of:
a current contention level of impacted data objects,
expected execution times of the competing transactions, and
transaction scopes of the competing transactions.
10 . The system of claim 1 , wherein the transactional database system is configured to:
generate a user interface to show pending transactions in the transactional database system; and
in response to user input via the user interface, kill one or more of the pending transactions.
11 . The system of claim 1 , wherein the transactional database system is configured to:
log a conflict history of transactions received by the transactional database system;
determine, based on the conflict history, a conflict hotspot within the file system; and
show the conflict hotspot via a user interface.
12 . The system of claim 1 , wherein the transactional database system is configured to:
determine a mitigation action to mitigate the conflict hotspot, to:
schedule a scheduled transactions to different time,
separate a future transaction into a set of smaller transactions, or
partition one or more data objects impacted by the conflict hotspot; and
show the mitigation action as a recommendation action via the user interface.
13 . A method, comprising:
executing a transactional database system implemented using one or more computer devices, wherein the execution comprises:
managing transactions for a database, wherein the database is stored in a file system as plurality of data objects, wherein individual data objects are files in the file system that store table data associated with respective tables of the database and individual tables of the database are stored under respective table directories in the file system;
initiating a first transaction that accesses a plurality of data items in the database, including creating a first copy of one or more first data objects to be accessed by the first transaction, wherein updates in the first transaction are performed on the first copy;
determining that a committed state of the one or more first data objects has not changed since the initiation of the first transaction, and in response:
committing the first transaction, including updating one or more metadata objects in the file system to refer to the first copy as the committed state of the one or more first data objects; and
initiating a second transaction that accesses the database, including creating a second copy of one or more second data objects to be accessed by the second transaction;
determining that a committed state of the one or more second data objects has changed since the initiation of the second transaction, and in response:
rolling back the second transaction, including abandoning the second copy of the one or more second data objects so that the second copy is not referred to as the committed state of the one or more second data objects; and
executing an asynchronous garbage collection process to remove copies of data objects that have been abandoned due to rollbacks, including the second copy of one or more second data objects.
14 . The method of claim 13 , wherein the second transaction is rolled back when a competing transaction that caused the change to the committed state of the one or more second data objects is committed.
15 . The method of claim 13 , further comprising the transactional database system:
receiving a conflict resolution policy via a configuration interface, wherein the conflict resolution policy specifies a conflict resolution option wherein a first finishing transaction in a group of competing transactions wins conflict resolution over one or more other transactions in the group.
16 . The method of claim 15 , wherein the conflict resolution policy specifies one or more rules to avoid starvation of transactions, wherein a particular transaction experiences starvation if it loses conflict resolution for a specified number of times.
17 . The method of claim 15 , wherein the conflict resolution policy selects a winning transaction from the group of competing transactions based on one or more of:
a current contention level of impacted data objects,
expected execution times of the competing transactions, and
transaction scopes of the competing transactions.
18 . The method of claim 13 , further comprising the transactional database system:
generating a user interface to show pending transactions in the transactional database system; and
in response to user input via the user interface, killing one or more of the pending transactions.
19 . The method of claim 13 , further comprising the transactional database system:
logging a conflict history of transactions received by the transactional database system;
determining, based on the conflict history, a conflict hotspot within the file system; and
showing the conflict hotspot via a user interface.
20 . A method of claim 19 , further comprising the transactional database system:
determining a mitigation action to mitigate the conflict hotspot, to:
schedule a scheduled transactions to different time,
separate a future transaction into a set of smaller transactions, or
partition one or more data objects impacted by the conflict hotspot; and
showing the mitigation action as a recommendation action via the user interface.