Decentralized database optimizations
Techniques for managing data stored within a database, such as a decentralized database are provided. Some techniques involve managing some data within a lower-trust database and some other data within a higher-trust database. A higher-trust database may be a decentralize database including a blockchain. A lower-trust database may store references to data within the blockchain, and optionally other data in association with those references. Disclosed techniques include WHERE clause query handling in databases with reference values, replacement of distinct data in a relational database with a distinct reference to that data, number line storing for secure indexing, APIs for databases, and consensus operations for private blockchain networks.
1. A tangible, non-transitory, machine-readable medium storing instructions that when executed by one or more processors effectuate operations comprising:
receiving, from a writing client application, a write request for a record to be stored in one or more blockchains of a blockchain-based datastore;
forming an identifier of the record that distinguishes the record from other records stored in the blockchain-based datastore;
providing the identifier of the record to the writing client application;
determining that the record is larger than a threshold size;
in response to the determination, encrypting received record data, storing resulting ciphertext in a cache datastore, and adding an entry to an index that maps the identifier to the ciphertext in the cache datastore;
storing at least part of the record in the blockchain-based datastore;
receiving, before storing at least one other part of the record in the blockchain-based datastore, a first read request from a first reading client application, the first reading application being the writing client application or a different client application;
in response to the first read request, accessing the ciphertext based on the entry in the index that maps the identifier to the ciphertext in the cache datastore, decrypting the ciphertext to obtain record data, and providing the obtained record data to the first reading client application;
updating, in response to storing the record in the blockchain-based datastore, the index to map the identifier to a location of a part of the record in a blockchain of the blockchain-based datastore and deleting a cryptographic key corresponding to the ciphertext or deleting the ciphertext in the cache datastore;
receiving, after storing the record in the blockchain-based datastore, a second read request from a second reading client application, the second reading application being the writing client application, the first reading client application, or a different client application; and
in response to the second read request, accessing the blockchain-based datastore based on the entry or an updated entry in the index that maps the identifier to the location of the part of the record in the blockchain of the blockchain-based datastore to obtain record data and providing the obtained record data to the second reading client application, wherein:
the write request, the first read request, and the second read request are representational state transfer application program interface requests, and
record data obtained from the part of the record is streamed to the second reading client application before at least one other part of the record containing other record data is accessed within the blockchain-based datastore.
2. The medium of claim 1 , wherein determining that the record is larger than a threshold size comprises:
detecting that the write request is indicative of the record being larger than a threshold size; and
determining to store received record data to the cache datastore prior to storage of the record the blockchain-based datastore.
3. The medium of claim 1 , wherein the threshold size is 800 KBs or greater.
4. The medium of claim 1 , wherein the threshold size is between:
1 KB and 800 KBs;
800 KBs and 1 MB; or
1 MB and 100 MBs.
5. The medium of claim 1 , wherein encrypting received record data and storing resulting ciphertext in a cache datastore comprises:
receiving a stream of record data from the writing client application;
generating the cryptographic key by which the record data is to be encrypted;
encrypting the stream of record data; and
storing the encrypted stream data in the cache datastore to store the resulting ciphertext of the record.
6. The medium of claim 1 , wherein storing at least part of the record in the blockchain-based datastore comprises:
detecting a first write of the record to the cache datastore is complete; and
initiating a second write of the record to the blockchain-based datastore, wherein the record within the cache datastore is maintained while performing the second write.
7. The medium of claim 1 , wherein storing at least part of the record in the blockchain-based datastore comprises:
setting a cursor to a first read position within record data, the first read position offset from an end of record data by a number of bytes;
reading the number of bytes from the first read position to the end of record data to obtain a last chunk of record data; and
storing the last chunk within the blockchain-based datastore.
8. The medium of claim 7 , further comprising:
setting the cursor to a second read position within record data, the second read position offset from the end of record data by the number of bytes plus a second number of bytes or offset from a first byte of the last chunk by a second number of bytes;
reading the second number of bytes from the second read position to the first read position to obtain a next last chunk of record data; and
storing the next last chunk within the blockchain-based datastore.
9. The medium of claim 8 , wherein the second number of bytes is the same as the first number of bytes.
10. The medium of claim 8 , further comprising:
storing the next last chunk within the blockchain-based datastore in association with information indicative of a location of the last chunk within the blockchain-based datastore.
11. The medium of claim 8 , further comprising:
obtaining and storing a plurality of additional chunks by setting the cursor within record data to read new bytes of record data until the cursor is set at a start of record data.
12. The medium of claim 11 , wherein:
the second number of bytes is the same as the first number of bytes, and the same number of bytes is read for at least some of the plurality of additional chunks.
13. The medium of claim 11 , wherein a last chunk obtained in the plurality of additional chunks comprises a given number of bytes read from the starts of record data to a preceding cursor position within the record data.
14. The medium of claim 1 , wherein:
storing the record in the blockchain-based datastore comprises storing a plurality of parts of the record with the blockchain-based datastore, each part having a respective location within a blockchain of the blockchain-based datastore;
the respective parts of the record are constituents of a linked list indicative of an ordered sequence in which to access parts to obtain the record; and
a next part of the record is accessed based on a location of the next part of the record being associated with a prior part of the record.
15. The medium of claim 14 , wherein:
the location of the next part of the record and a location of the prior part of the record are not sequential values.
16. The medium of claim 14 , wherein:
the next part of the record and the prior part of the record for at least some adjacent parts of the record are not sequential transactions in a central queue of transactions for the blockchain-based datastore.
17. The medium of claim 14 , wherein:
a respective location of a respective part of the record within the blockchain-based datastore is a respective hash pointer to a location within a blockchain of the blockchain-based datastore.
18. The medium of claim 1 , further comprising:
scattering the different parts of the record within the blockchain-based datastore.
19. The medium of claim 1 , wherein:
the blockchain-based datastore comprises at least two blockchains.
20. The medium of claim 1 , wherein accessing the blockchain-based datastore based on the entry or an updated entry in the index that maps the identifier to the location of the part of the record in the blockchain of the blockchain-based datastore to obtain record data comprises:
accessing the part of the record in the blockchain based on the location and to obtain first record data;
obtaining a location of a next part of the record indicated by the part of the record; and
accessing the next part of the record based on the obtained location for the next part to obtain second record data.
21. The medium of claim 20 , wherein the first record data and the second record data are sequential portions of the record.
22. The medium of claim 1 , further comprising:
providing a client-facing representational state transfer application program interface; and
receiving the write request, the first read request, and the second read request via the interface,
wherein the cache datastore is a persistent storage bucket distinct from the blockchain-based datastore.
23. A computer-implemented method comprising:
receiving, from a writing client application, a write request for a record to be stored in one or more blockchains of a blockchain-based datastore;
forming an identifier of the record that distinguishes the record from other records stored in the blockchain-based datastore;
providing the identifier of the record to the writing client application;
determining that the record is larger than a threshold size;
in response to the determination, encrypting received record data, storing resulting ciphertext in a cache datastore, and adding an entry to an index that maps the identifier to the ciphertext in the cache datastore;
storing at least part of the record in the blockchain-based datastore;
receiving, before storing at least one other part of the record in the blockchain-based datastore, a first read request from a first reading client application, the first reading application being the writing client application or a different client application;
in response to the first read request, accessing the ciphertext based on the entry in the index that maps the identifier to the ciphertext in the cache datastore, decrypting the ciphertext to obtain record data, and providing the obtained record data to the first reading client application;
updating, in response to storing the record in the blockchain-based datastore, the index to map the identifier to a location of a part of the record in a blockchain of the blockchain-based datastore and deleting a cryptographic key corresponding to the ciphertext or deleting the ciphertext in the cache datastore;
receiving, after storing the record in the blockchain-based datastore, a second read request from a second reading client application, the second reading application being the writing client application, the first reading client application, or a different client application; and
in response to the second read request, accessing the blockchain-based datastore based on the entry or an updated entry in the index that maps the identifier to the location of the part of the record in the blockchain of the blockchain-based datastore to obtain record data and providing the obtained record data to the second reading client application, wherein:
the write request, the first read request, and the second read request are representational state transfer application program interface requests, and
record data obtained from the part of the record is streamed to the second reading client application before at least one other part of the record containing other record data is accessed within the blockchain-based datastore.
24. The method of claim 23 , wherein determining that the record is larger than a threshold size comprises:
detecting that the write request is indicative of the record being larger than a threshold size; and
determining to store received record data to the cache datastore prior to storage of the record the blockchain-based datastore.
25. The method of claim 23 , wherein the threshold size is 800 KBs or greater.
26. The method of claim 23 , wherein the threshold size is between:
1 KB and 800 KBs;
800 KBs and 1 MB; or
1 MB and 100 MBs.
27. The method of claim 23 , wherein encrypting received record data and storing resulting ciphertext in a cache datastore comprises:
receiving a stream of record data from the writing client application;
generating the cryptographic key by which the record data is to be encrypted;
encrypting the stream of record data; and
storing the encrypted stream data in the cache datastore to store the resulting ciphertext of the record.
28. The method of claim 23 , wherein storing at least part of the record in the blockchain-based datastore comprises:
detecting a first write of the record to the cache datastore is complete; and
initiating a second write of the record to the blockchain-based datastore, wherein the record within the cache datastore is maintained while performing the second write.
29. The method of claim 23 , wherein storing at least part of the record in the blockchain-based datastore comprises:
setting a cursor to a first read position within record data, the first read position offset from an end of record data by a number of bytes;
reading the number of bytes from the first read position to the end of record data to obtain a last chunk of record data; and
storing the last chunk within the blockchain-based datastore.
30. The method of claim 29 , further comprising:
setting the cursor to a second read position within record data, the second read position offset from the end of record data by the number of bytes plus a second number of bytes or offset from a first byte of the last chunk by a second number of bytes;
reading the second number of bytes from the second read position to the first read position to obtain a next last chunk of record data; and
storing the next last chunk within the blockchain-based datastore.
31. The method of claim 30 , wherein the second number of bytes is the same as the first number of bytes.
32. The method of claim 30 , further comprising:
storing the next last chunk within the blockchain-based datastore in association with information indicative of a location of the last chunk within the blockchain-based datastore.
33. The method of claim 30 , further comprising:
obtaining and storing a plurality of additional chunks by setting the cursor within record data to read new bytes of record data until the cursor is set at a start of record data.
34. The method of claim 33 , wherein:
the second number of bytes is the same as the first number of bytes, and the same number of bytes is read for at least some of the plurality of additional chunks.
35. The method of claim 33 , wherein a last chunk obtained in the plurality of additional chunks comprises a given number of bytes read from the starts of record data to a preceding cursor position within the record data.
36. The method of claim 23 , wherein:
storing the record in the blockchain-based datastore comprises storing a plurality of parts of the record with the blockchain-based datastore, each part having a respective location within a blockchain of the blockchain-based datastore;
the respective parts of the record are constituents of a linked list indicative of an ordered sequence in which to access parts to obtain the record; and
a next part of the record is accessed based on a location of the next part of the record being associated with a prior part of the record.
37. The method of claim 36 , wherein:
the location of the next part of the record and a location of the prior part of the record are not sequential values.
38. The method of claim 36 , wherein:
the next part of the record and the prior part of the record for at least some adjacent parts of the record are not sequential transactions in a central queue of transactions for the blockchain-based datastore.
39. The method of claim 36 , wherein:
a respective location of a respective part of the record within the blockchain-based datastore is a respective hash pointer to a location within a blockchain of the blockchain-based datastore.
40. The method of claim 23 , further comprising:
scattering the different parts of the record within the blockchain-based datastore.
41. The method of claim 23 , wherein:
the blockchain-based datastore comprises at least two blockchains.
42. The method of claim 23 , wherein accessing the blockchain-based datastore based on the entry or an updated entry in the index that maps the identifier to the location of the part of the record in the blockchain of the blockchain-based datastore to obtain record data comprises:
accessing the part of the record in the blockchain based on the location and to obtain first record data;
obtaining a location of a next part of the record indicated by the part of the record; and
accessing the next part of the record based on the obtained location for the next part to obtain second record data.
43. The method of claim 42 , wherein the first record data and the second record data are sequential portions of the record.
44. The method of claim 23 , further comprising:
providing a client-facing representational state transfer application program interface; and
receiving the write request, the first read request, and the second read request via the interface,
wherein the cache datastore is a persistent storage bucket distinct from the blockchain-based datastore.