Architecture of immutable database for bitemporal analysis
Embodiments disclosed are directed to implementing an immutable database. Several embodiments are directed to an architecture for implementing the immutable database as a data processing system with a real-time interface for buffering a plurality of transactions and a bitemporal ledger for storing the plurality of transactions and a processor. New transactions may be appended to the transaction to the plurality of transactions and processed in a temporal order which includes identifying a sequence of transactions in the bitemporal ledger associated with the transaction and incorporating the transaction into the sequence of transactions in the bitemporal ledger according to temporal metadata associated with the transaction to form an updated sequence of transactions.
1 . A data processing system comprising:
a data stream manager configured to:
generate a data stream associated with a data stream identifier, wherein the data stream identifier identifies a user account managed by the data processing system;
receive, in the data stream, a plurality of transactions including a new transaction and a retroactive transaction, wherein the new transaction and the retroactive transaction are associated with the user account;
a processing engine configured to receive, from the data stream manager, the plurality of transactions including the new transaction and the retroactive transaction, and wherein the processing engine is further configured to:
identify, in a bitemporal ledger in communication with the processing engine, a sequence of transactions associated with the new transaction and the retroactive transaction and an alternate sequence of transactions, wherein the sequence of transactions is implemented as a transaction timeline associated with the user account, and wherein the alternate sequence of transactions is implemented as an alternate timeline associated with the user account;
append, in the bitemporal ledger, the new transaction to the sequence of transactions according to temporal metadata associated with the new transaction, wherein the temporal metadata includes a first time indicating when the new transaction is created and a second time indicating when the new transaction is to take effect, wherein the bitemporal ledger is configured to prevent any transactions in the sequence of transactions from being overwritten during the appending, wherein the transaction timeline comprising the sequence of transactions and the new transaction reflects a current state of the user account, and wherein the sequence of transactions is time-ordered based at least in part on the second time of the temporal metadata; and
incorporate, in the bitemporal ledger, the retroactive transaction to the alternate sequence of transactions by replacing one transaction in the sequence of transactions in the alternate timeline with the retroactive transaction; and
the bitemporal ledger configured to store the transaction timeline and the alternate timeline, wherein the bitemporal ledger is configured to:
maintain the alternate timeline separate from the transaction timeline, wherein the alternate sequence of transactions reflects an alternate state of the user account based on the retroactive transaction, and wherein the alternate sequence of transactions is time-ordered based at least in part on temporal metadata associated with the retroactive transaction;
determine whether the alternate state of the user account includes an error, wherein the evaluation comprises a comparison between the current state and the alternate state; and
merge, in the transaction timeline, the current state of the user account with the alternate state of the user account based on the determining.
2 . The data processing system of claim 1 ,
wherein the transaction timeline further comprises a second sequence of transactions, wherein the second sequence of transactions is associated with a second user account managed by the data processing system.
3 . The data processing system of claim 1 , wherein the sequence of transactions is implemented as a graph comprising a plurality of nodes with each node of the plurality of nodes having a corresponding transaction in the sequence of transactions.
4 . The data processing system of claim 1 , the processing engine further configured to:
identify a plug-in configured to inspect the new transaction for validity; and
pass the new transaction to the plug-in for inspecting the new transaction for validity.
5 . The data processing system of claim 4 , wherein the alternate sequence of transactions further includes alternate transactions that are in temporal order subsequent to the sequence of transactions.
6 . The data processing system of claim 5 , wherein at least one alternate transaction of the alternate transactions reflects a change to the sequence of transactions and is based on the new transaction.
7 . The data processing system of claim 4 , the processing engine is further configured to:
generate an event that indicates the new transaction is ready for further processing when the new transaction is valid; and
generate an error message when the new transaction is not valid.
8 . The data processing system of claim 1 , wherein the new transaction includes an identifier associated with the sequence of transactions and wherein identifying the sequence of transactions is based on the identifier.
9 . A computer-implemented method for processing a new transaction in a data processing system that includes a real-time interface, a processing engine, and a bitemporal ledger, the method comprising:
generating a data stream associated with a data stream identifier, wherein the data stream identifier identifies a user account managed by the data processing system;
receiving, in the data stream via the real-time interface, the new transaction and a retroactive transaction to be processed by the data processing system, wherein the real-time interface comprises a transaction stream for buffering a plurality of transactions and wherein the new transaction and the retroactive transaction are associated with the user account managed by the data processing system;
identifying, in the bitemporal ledger in communication with the processing engine, a sequence of transactions associated with the new transaction and the retroactive transaction and an alternate sequence of transactions, wherein the sequence of transactions is implemented as a transaction timeline associated with the user account, and wherein the alternate sequence of transactions is implemented as an alternate timeline associated with the user account;
appending, in the bitemporal ledger, the new transaction to the sequence of transactions according to temporal metadata associated with the new transaction, wherein the temporal metadata includes a first time indicating when the new transaction is created and a second time indicating when the new transaction is to take effect, wherein the bitemporal ledger is configured to prevent any transactions in the sequence of transactions from being overwritten during the appending, wherein the transaction timeline comprising the sequence of transactions and the new transaction reflects a current state of the user account, and wherein the sequence of transactions is time-ordered based at least in part on the second time of the temporal metadata;
incorporating, in the bitemporal ledger, the retroactive transaction to the alternate sequence of transactions by replacing one transaction in the alternate sequence of transactions with the retroactive transaction;
maintaining the alternate timeline separate from the transaction timeline in the bitemporal ledger, wherein the alternate sequence of transactions reflects an alternate state of the user account based on the retroactive transaction, and wherein the alternate sequence of transactions is time-ordered based at least in part on temporal metadata associated with the retroactive transaction;
determining whether the alternate state of the user account includes an error; and
merging, in the transaction timeline, in the bitemporal ledger, the current state of the user account with the alternate state of the user account based on the determining.
10 . The method of claim 9 , wherein the transaction timeline further comprises a second sequence of transactions, wherein the second sequence of transactions is associated with a second user account managed by the data processing system.
11 . The method of claim 9 , wherein the sequence of transactions is implemented as a graph comprising a plurality of nodes with each node of the plurality of nodes having a corresponding transaction in the sequence of transactions.
12 . The method of claim 9 , further comprising:
identifying, by the processing engine, a processing module configured to further process the new transaction; and
passing, by the processing engine, the new transaction to the processing module for further processing.
13 . The method of claim 12 , wherein the alternate sequence of transactions further includes alternate transactions that are in temporal order subsequent to the sequence of transactions.
14 . The method of claim 13 , wherein at least one alternate transaction of the alternate transactions reflects a change to the sequence of transactions and is based on the new transaction.
15 . The method of claim 12 , further comprising:
storing, by the processing engine, results of the further processing by the processing module in the bitemporal ledger.
16 . The method of claim 9 , wherein the new transaction includes an identifier associated with the sequence of transactions and wherein identifying the sequence of transactions is based on the identifier.
17 . A non-transitory computer-readable medium storing instructions, the instructions, when executed by an immutable database that includes a processing engine, a real-time interface and a bitemporal ledger, cause the immutable database to perform operations comprising:
generating a data stream associated with a data stream identifier, wherein the data stream identifier identifies a user account managed by the data processing system;
receiving, in the data stream via the real-time interface, a new transaction and a retroactive transaction to be processed by the data processing system, wherein the real-time interface comprises a transaction stream for buffering a plurality of transactions and wherein the new transaction and the retroactive transaction are associated with the user account managed by the data processing system;
identifying, in the bitemporal ledger in communication with the processing engine, a sequence of transactions associated with the new transaction and the retroactive transaction and an alternate sequence of transactions, wherein the sequence of transactions is implemented as a transaction timeline associated with the user account, and wherein the alternate sequence of transactions is implemented as an alternate timeline associated with the user account;
appending, in the bitemporal ledger, the new transaction to the sequence of transactions according to temporal metadata associated with the new transaction, wherein the temporal metadata includes a first time indicating when the new transaction is created and a second time indicating when the new transaction is to take effect, wherein the bitemporal ledger is configured to prevent any transactions in the sequence of transactions from being overwritten, wherein the transaction timeline comprising the sequence of transactions and the new transaction reflects a current state of the user account, and wherein the sequence of transactions is time-ordered based at least in part on the second time of the temporal metadata;
incorporating, in the bitemporal ledger, the retroactive transaction to the alternate sequence of transactions by replacing one transaction in the alternate sequence of transactions with the retroactive transaction;
maintaining the alternate timeline separate from the transaction timeline in the bitemporal ledger, wherein the alternate sequence of transactions reflects an alternate state of the user account based on the retroactive transaction, and wherein the alternate sequence of transactions is time-ordered based at least in part on temporal metadata associated with the retroactive transaction;
determining whether the alternate state of the user account includes an error; and
merging, in the transaction timeline, in the bitemporal ledger, the current state of the user account with the alternate state of the user account based on the determining.
18 . The non-transitory computer-readable medium claim 17 , the operations further comprising:
storing, in the bitemporal ledger, a second sequence of transactions, wherein the second sequence of transactions is associated with a second user account managed by the data processing system.
19 . The non-transitory computer-readable medium of claim 17 , wherein the sequence of transactions is implemented as a graph comprising a plurality of nodes with each node of the plurality of nodes having a corresponding transaction in the sequence of transactions.
20 . The non-transitory computer-readable medium of claim 17 , the operations further comprising:
identifying, by the processing engine, a processing module configured to further process the new transaction; and
passing, by the processing engine, the new transaction to the processing module for further processing.