Query optimized distributed ledger system
Disclosed herein are distributed ledger systems and methods for efficiently creating and updating a query optimized distributed ledger. In particular, the present disclosure introduces methods and apparatuses for efficiently updating indexes when new blocks are added to the distributed ledger by using snapshots of data and appending new snapshot tables and indexes to previous snapshot tables and indexes.
1. A method for indexing a distributed ledger, the method comprising:
receiving, with a hardware processor of a data node, a first snapshot of transaction data, wherein the first snapshot of transaction data is data added to the distributed ledger that has not been included in an original master table or an original index of the distributed ledger;
identifying, with a hardware processor of a data node, attributes of the transaction data of the first snapshot; verifying, with the hardware processor, the first snapshot;
copying, with a hardware processor of a data node, the attributes of the transaction data of the first snapshot to a first master table;
constructing, with a hardware processor of a data node, a first index for a first attribute of the transaction data of the first snapshot;
publishing, with a hardware processor of a data node, completion of the first index for the first attribute of the transaction data of the first snapshot;
concatenating the original master table and the first master table;
concatenating the original index and the first index;
receiving a request to query the distributed ledger transaction data; and
processing the query on the indexed attributes.
2. The method of claim 1 , further comprising:
constructing a first index for a second attribute of the transaction data of the first snapshot; and
publishing completion of the first index for the second attribute.
3. The method of claim 2 , further comprising:
repeating the operation of constructing a first index and publishing completion of the first index for each identified attribute of the first snapshot.
4. The method of claim 1 , further comprising:
receiving, with the hardware processor, a second snapshot of transaction data, wherein the second snapshot of transaction data is data added to the distributed ledger that has not been included in the original or first master tables or the original or first indexes of the distributed ledger;
identifying, with the hardware processor, attributes of the transaction data of the second snapshot;
verifying, with the hardware processor, the second snapshot;
copying, with the hardware processor, the attributes of the transaction data of the second snapshot to a second master table;
constructing, with the hardware processor, a second index for the first attribute of the transaction data of the second snapshot; and
publishing, with the hardware processor, completion of the second index for the first attribute of the transaction data of the second snapshot.
5. The method of claim 4 , further comprising:
concatenating the concatenated master table of claim 1 and the second master table; and
concatenating the concatenated index of claim 1 and the second index.
6. The method of claim 5 , further comprising:
receiving, with the hardware processor, a third snapshot of transaction data, wherein the third snapshot of transaction data is data added to the distributed ledger that has not been included in the original, first, or second master tables or the original, first, or second indexes of the distributed ledger;
identifying, with the hardware processor, attributes of the transaction data of the third snapshot;
verifying, with the hardware processor, the third snapshot;
copying, with the hardware processor, the attributes of the transaction data of the third snapshot to a third master table;
constructing, with the hardware processor, a third index for the first attribute of the transaction data of the third snapshot; and
publishing, with the hardware processor, completion of the third index for the first attribute of the transaction data of the third snapshot.
7. The method of claim 6 , further comprising:
concatenating the concatenated master table of claim 5 and the third master table; and
concatenating the concatenated index of claim 5 and the third index for the first attribute.
8. The method of claim 4 , further comprising:
repeating the operation of constructing a second index and publishing completion of the second index for each identified attribute of the second snapshot.
9. The method of claim 8 , further comprising:
concatenating the concatenated index of claim 5 and the second index for each indexed attribute.
10. The method of claim 4 , wherein verifying the transaction data is performed by calculating the delta between the Merkle root of the first snapshot and the Merkle root of the second snapshot.
11. The method of claim 1 , wherein executing the query on the indexed attributes is performed on a plurality of data nodes using distributed query processing.
12. The method of claim 1 , wherein verifying the transaction data is performed by calculating the Merkle root.
13. A system for indexing a distributed ledger, the system comprising:
a transaction node, the transaction node comprising:
memory; and
a hardware processor, wherein the hardware processor of the transaction node is configured to:
broadcast a block of transaction data; and
a data node, the data node comprising:
memory;
and a hardware processor, wherein the hardware processor of the data node is configured to:
receive a first snapshot of transaction data, wherein the first snapshot of transaction data is data added to the distributed ledger that has not been included in a master table or an index of the distributed ledger;
identify attributes of the transaction data of the first snapshot;
verify the first snapshot;
copy the attributes of the transaction data of the first snapshot to a first master table;
construct a first index for a first attribute of the transaction data of the first snapshot; and
publish completion of the first index for the first attribute of the transaction data of the first snapshot;
concatenate the original master table and the first master table;
concatenate the original index and the first index;
receive a request to query the distributed ledger transaction data; and
process the query on the indexed attributes.
14. The system of claim 13 , wherein the hardware processor is further configured to:
repeat the operation of constructing a first index and publishing completion of the first index for each identified attribute of the transaction data of the first snapshot.
15. The system of claim 13 , wherein the hardware processor is further configured to:
receive a second snapshot of transaction data;
identify attributes of the transaction data of the second snapshot;
verify the second snapshot;
copy the attributes of the transaction data of the second snapshot to a second master table;
construct a second index for the first attribute of the transaction data of the second snapshot; and
publish completion of the second index for the first attribute of the second snapshot.
16. The system of claim 15 , wherein the hardware processor is further configured to:
concatenate the first master table and the second master table; and
concatenate the first index and the second index.
17. The system of claim 15 , wherein the hardware processor is further configured to:
repeat the operation of constructing a second index and publishing completion of the second index for each identified attribute of the second snapshot.
18. The system of claim 15 , wherein the hardware processor is further configured to:
repeat the operation of constructing a second index and publishing completion of the second index for each identified attribute of the second snapshot, wherein each data node is assigned to start indexing at a different identified attribute of second snapshot.
19. The system of claim 13 , wherein the hardware processor is further configured to:
repeat the operation of constructing a first index and publishing completion of the first index for each identified attribute of the first snapshot, wherein each data node is assigned to start indexing at a different identified attribute of the first snapshot.
20. One or more machine-readable storage media comprising one or more instructions stored thereon that when executed by at least one processor, cause the at least one processor to:
receive, with a hardware processor of a data node, a first snapshot of transaction data, wherein the first snapshot of transaction data is data added to the distributed ledger that has not been included in an original master table or an original index of the distributed ledger;
identify, with a hardware processor of a data node, attributes of the transaction data of the first snapshot; verifying, with the hardware processor, the first snapshot;
copy, with a hardware processor of a data node, the attributes of the transaction data of the first snapshot to a first master table;
construct, with a hardware processor of a data node, a first index for a first attribute of the transaction data of the first snapshot;
publish, with a hardware processor of a data node, completion of the first index for the first attribute of the transaction data of the first snapshot;
concatenate the original master table and the first master table;
concatenate the original index and the first index;
receive a request to query the distributed ledger transaction data; and
process the query on the indexed attributes.
21. The at least one machine readable medium of claim 20 , further comprising one or more instructions that when executed by the at least one processor, cause the at least one processor to:
receive a second snapshot of transaction data from the distributed ledger;
identify attributes of the transaction data of the second snapshot;
verify the second snapshot;
copy the attributes of the transaction data of the second snapshot to a second master table;
construct a second index for the first attribute of the second snapshot; and
publish completion of the second index for the first attribute of the second snapshot.
22. The media of claim 21 , wherein the instructions are further adapted to cause the at least one processor to:
concatenate the first master table and the second master table; and
concatenate the first index and the second index.