Systems and methods for a scalable versioned database
Systems and methods that for a scalable versioned database that can organize data into scenarios and hives. These define how data visibility is controlled by scenarios, and how data can be transferred between scenarios. In some embodiments, queries and algorithms can be executed in independent processes, which may execute in parallel, and on independent machines. Furthermore, data objects can be placed in shared storage, and metadata objects can be placed in a metadata database which supports ACID transactions. Data objects are not modified in place after they are constructed, while metadata objects can be updated transactionally, using the metadata database. Data can be updated by creating new data objects and connecting them to scenarios with new metadata objects. Finally, obsolete data may be cleaned up by identifying unreachable data, which is safe to delete.
1 . A computing apparatus comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the apparatus to:
open, by the processor, a transaction in a metadata key-value database;
obtain, by the processor, a scenario definition of a scenario from the metadata key-value database, the scenario definition comprising a current snapshot ID and a parent or root scenario ID;
obtain, by the processor, the current snapshot ID from the scenario definition;
add, by the processor, a view in the metadata key-value database, the view referencing the current snapshot ID;
commit, by the processor, the transaction in the metadata key-value database;
execute, by the processor, a read query;
open, by the processor, the transaction in the metadata key-value database;
remove, by the processor, the view in the metadata key-value database; and
commit, by the processor, the transaction in the metadata key-value database.
2 . The computing apparatus of claim 1 , wherein when executing the read query, the apparatus is further configured to accessing snapshot data.
3 . The computing apparatus of claim 2 , wherein when accessing the snapshot data, the apparatus is further configured to:
obtain, by the processor, a schema ID from a snapshot corresponding to the current snapshot ID;
obtain, by the processor, a schema corresponding to the schema ID from a schema storage object;
obtain, by the processor, a location of a column instance ID in the snapshot;
obtain, by the processor, the column instance ID using the location; and
obtain, by the processor, a column instance from a data object storage, using the column instance ID.
4 . The computing apparatus of claim 1 , wherein the transaction is committed in the metadata key-value database provided the scenario definition remained unchanged outside the transaction since the transaction opened.
5 . A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:
open, by a processor, a transaction in a metadata key-value database;
obtain, by the processor, a scenario definition of a scenario from the metadata key-value database, the scenario definition comprising a current snapshot ID and a parent or root scenario ID;
obtain, by the processor, the current snapshot ID from the scenario definition;
add, by the processor, a view in the metadata key-value database, the view referencing the current snapshot ID;
commit, by the processor, the transaction in the metadata key-value database;
execute, by the processor, a read query;
open, by the processor, the transaction in the metadata key-value database;
remove, by the processor, the view in the metadata key-value database; and
commit, by the processor, the transaction in the metadata key-value database.
6 . The computer-readable storage medium of claim 5 , wherein when executing the read query, the computer is further configured to access snapshot data.
7 . The computer-readable storage medium of claim 6 , wherein when accessing the snapshot data, the computer is further configured to:
obtain, by the processor, a schema ID from a snapshot corresponding to the current snapshot ID;
obtain, by the processor, a schema corresponding to the schema ID from a schema storage object;
obtain, by the processor, a location of a column instance ID in the snapshot;
obtain, by the processor, the column instance ID using the location; and
obtain, by the processor, a column instance from a data object storage, using the column instance ID.
8 . The computer-readable storage medium of claim 5 , wherein the transaction is committed in the metadata key-value database provided the scenario definition remained unchanged outside the transaction since the transaction opened.
9 . A computer-implemented method for executing a read query in a scenario in a scalable versioned database, the method comprising:
opening, by a processor, a transaction in a metadata key-value database;
obtaining, by the processor, a scenario definition of the scenario from the metadata key-value database, the scenario definition comprising a current snapshot ID and a parent or root scenario ID;
obtaining, by the processor, the current snapshot ID from the scenario definition;
adding, by the processor, a view in the metadata key-value database, the view referencing the current snapshot ID;
committing, by the processor, the transaction in the metadata key-value database;
executing, by the processor, the read query;
opening, by the processor, the transaction in the metadata key-value database;
removing, by the processor, the view in the metadata key-value database; and
committing, by the processor, the transaction in the metadata key-value database.
10 . The computer-implemented method of claim 9 , wherein executing the read query comprises accessing snapshot data.
11 . The computer-implemented method of claim 10 , wherein accessing the snapshot data comprises:
obtaining, by the processor, a schema ID from a snapshot corresponding to the current snapshot ID;
obtaining, by the processor, a schema corresponding to the schema ID from a schema storage object;
obtaining, by the processor, a location of a column instance ID in the snapshot;
obtaining, by the processor, the column instance ID using the location; and
obtaining, by the processor, a column instance from a data object storage, using the column instance ID.
12 . The computer-implemented method of claim 9 , wherein the transaction is committed in the metadata key-value database provided the scenario definition remained unchanged outside the transaction since the transaction opened.