Methods and systems for resilient, durable, scalable, and consistent distributed timeline data store
The present disclosure discloses methods and systems for managing data in a database in accordance with a data model, henceforth referred to as a “timeline store” or “timeline model”. The method includes mapping each data reference of one or more data references with a data value of a block of data in a data store. Then, each key reference of one or more key references is mapped with at least one pair of a time reference and the data reference in a timeline store, the time reference determines a point in time at which the key reference is assigned the pair of the time reference and the data reference. Thereafter, the data model is queried to retrieve a set of key references mapped to data references based on a specific ranges of keys and time or time intervals. Finally, the data model is queried to retrieve the raw data bytes associated/mapped with/to any desired data reference.
1. A method for managing data in a database in accordance with a temporal data model, the method comprising:
mapping each data reference of one or more data references with a data value of a block of data in a data store, using a data update module;
mapping each key reference of one or more key references with at least one pair of a time reference and the data reference in a timeline store, using the data update module, wherein the time reference determines a point in time at which the key reference is assigned the pair of the time reference and the data reference, and wherein the key reference is a time specific pointer identifying a particular data reference at a particular time;
modelling the timeline store as a plurality of data triplets, each data triplet including one key reference, one time reference, and one data reference;
querying the data model to retrieve a set of key references mapped to data references based on a specific range of time, using a query processor;
querying the data model to retrieve a set of data references based on a specific range of time, using the query processor; and
mapping the key reference in the timeline store to a new pair of a time reference and a data reference when the key reference is updated,
wherein updates to the key reference are managed using a distributed and fault-tolerant agreement algorithm implemented using a distributed set of RAFT or PAXOS groups, wherein each of the RAFT or PAXOS groups runs on one or more nodes.
2. The method for managing data of claim 1 , further comprising querying the data model to retrieve a set of key references mapped to data references based on a specific range of keys.
3. The method for managing data of claim 1 , further comprising querying the data model to retrieve a set of data references based on a specific range of keys.
4. The method for managing data of claim 1 , further comprising querying the data model to retrieve a set of key references mapped to data references based on a specific combined range of time and keys.
5. The method for managing data of claim 1 , further comprising querying the data model to retrieve a set of data references based on a specific combined range of time and keys.
6. The method for managing data of claim 1 , further comprising loading the timeline store in a main memory for query processing.
7. The method for managing data of claim 1 , wherein updates to the key reference are managed based on a pre-defined condition.
8. The method for managing data of claim 7 , wherein the pre-defined condition is to check if the update is within the key-time range specified in a transaction request.
9. The method for managing data of claim 1 , further comprising deduplicating data in the data store.
10. The method for managing data of claim 9 further comprising deduplicating data in the data store based on exact matching of content of the block of data.
11. A database management system for managing data in accordance with a temporal data model, the database management system comprising:
a timeline store configured to store one or more key references, wherein each key reference of the one or more key references is mapped to at least one pair of a time reference and a data reference, and wherein the time reference determines a point in time at which the key reference is assigned the pair of the time reference and the data reference, and wherein the key reference is a time specific pointer identifying a particular data reference at a particular time, and the timeline store is modelled as a plurality of data triplets, each data triplet including one key reference, one time reference, and one data reference;
a data store configured to store one or more data references, wherein each data reference is mapped to a data value;
a query processor configured to:
query the data model to retrieve a set of key references mapped to data references based on a specific range of time; and
query the data model to retrieve a set of data references based on a specific range or ranges of time; and
a data update module configured to map the key reference in the timeline store to a new pair of a time reference and a data reference when the key reference is updated,
wherein updates to the key reference are managed using a distributed and fault-tolerant agreement algorithm implemented using a distributed set of RAFT or PAXOS groups, wherein each of the RAFT or PAXOS groups runs on one or more nodes.
12. The database management system of claim 11 , wherein the query processor is further configured to query the data model based on at least one of a range of key references and a range of time references.
13. The database management system of claim 11 , further comprising a data deduplication module configured to deduplicate data in the data store.
14. The database management system of claim 11 , further comprising a time management module configured to use a hybrid time for event time resolution, wherein the hybrid time is based on system clock time and an additional data set.
15. The database management system of claim 11 , further comprising a transaction processing module configured to implement a distributed set of consensus processing groups, wherein each of the consensus processing groups runs on one or more nodes.