Methods and systems for real-time continuous updates
Embodiments of the present invention provide fine grain concurrency control for transactions in the presence of database updates. During operations, each transaction is assigned a snapshot version number or SVN. A SVN refers to a historical snapshot of the database that can be created periodically or on demand. Transactions are thus tied to a particular SVN, such as, when the transaction was created. Queries belonging to the transactions can access data that is consistent as of a point in time, for example, corresponding to the latest SVN when the transaction was created. At various times, data from the database stored in a memory can be updated using the snapshot data corresponding to a SVN. When a transaction is committed, a snapshot of the database with a new SVN is created based on the data modified by the transaction and the snapshot is synchronized to the memory. When a transaction query requires data from a version of the database corresponding to a SVN, the data in the memory may be synchronized with the snapshot data corresponding to that SVN.
1. A method of enabling queries to continuously operate on different states of data while allowing the data to be manipulated, said method comprising:
receiving queries and transactions on data simultaneously in the database, wherein the queries operate on a data stored in a first memory coupled to a hardware accelerator and the transactions operate on another data stored in a second memory coupled to a host;
taking snapshots of data in the second memory at an interval;
determining a set of changes that takes the data in the first memory from one snapshot to another snapshot;
generating a mask comprising the set of changes that is applied by a machine code database instruction for changing the data in the first memory from one snapshot to the other snapshot;
determining, for each query, a snapshot that is associated with that query; and
determining data in the first memory on which the queries will operate based on the snapshot and selectively applying the mask.
2. The method of claim 1 , wherein determining the data on which the queries will operate comprises determining when data on which the queries will operate is absent from the first memory and prefetching the data into the first memory when the data is absent.
3. The method of claim 1 , wherein determining the data on which the queries will operate comprises determining when data on which the queries will operate are present in the first memory and changing the data from one snapshot to another snapshot based on the mask.
4. A system that enables queries to operate on different states of data from a database while allowing transactions on the data, wherein the data can be stored in a first memory and a second memory, said system comprising:
a query engine that performs queries on data stored in a first memory;
a transaction engine that performs transactions on at least some of the same data stored in a second memory during the same time that the queries are performed; and
a transaction manager configured to determine snapshots of the data in the second memory at an interval, determine a set of changes that tracks changes in the data in the second memory from one snapshot to another snapshot, and modify data in the first memory on which the queries will operate based on snapshots assigned to each of the queries and selectively requesting a machine code database instruction that applies the changes to the data in the first memory.
5. The system of claim 4 , wherein the transaction manager is configured to determine when data on which the queries will operate is absent from the first memory and prefetch the data into the first memory when the data is absent.
6. The system of claim 4 , wherein the transaction manager is configured to determine when data on which the queries will operate are present in the first memory and changing the data from one snapshot to another snapshot based on the mask.
7. A non-transitory tangible computer readable medium comprising program code for performing a method of enabling queries to continuously operate on different states of data while allowing the data to be manipulated, said computer readable medium comprising:
program code for receiving queries and transactions on data simultaneously in the database, wherein the queries operate on a data stored in a first memory coupled to a hardware accelerator and the transactions operate on another data stored in a second memory coupled to a host;
program code for taking snapshots of data in the second memory at an interval; program code for determining a set of changes that takes the data in the first memory from one snapshot to another snapshot;
program code for generating a mask comprising the set of changes that is applied by a machine code database instruction for changing the data in the first memory from one snapshot to the other snapshot, program code for determining, for each query, a snapshot that is associated with that query; and
program code for determining data on which the queries will operate based on the snapshot and selectively applying the mask.
8. The computer readable medium of claim 7 , wherein the program code for determining the data on which the queries will operate comprises program code for determining when data on which the queries will operate is absent from the first memory and program code for prefetching the data into the first memory when the data is absent.
9. The computer readable medium of claim 7 , wherein the program code for determining the data on which the queries will operate comprises program code for determining when data on which the queries will operate are present in the first memory and program code for changing the data from one snapshot to another snapshot based on the mask.
10. A method of reconstructing data for a transaction that has been interrupted, said method comprising:
receiving queries and transactions on data simultaneously in the database, wherein the queries operate on data stored in a first memory coupled to a hardware accelerator and the transactions operate on another copy of the data stored in a second memory coupled to a host;
taking snapshots of data in the second memory at an interval;
determining a set of changes that takes the data in the first memory from one snapshot to another snapshot;
detecting an interruption in the transaction;
determining a snapshot in which the interruption occurred;
generating a mask comprising the set of changes that is applied by a machine code database instruction for changing the data from the first memory from one snapshot to the snapshot in which the interruption occurred; and
recovering data from the first memory into the second memory based on the mask.