Decentralized database optimizations
Techniques for managing data stored within a database, such as a decentralized database are provided. Some techniques involve managing some data within a lower-trust database and some other data within a higher-trust database. A higher-trust database may be a decentralize database including a blockchain. A lower-trust database may store references to data within the blockchain, and optionally other data in association with those references. Disclosed techniques include WHERE clause query handling in databases with reference values, replacement of distinct data in a relational database with a distinct reference to that data, number line storing for secure indexing, APIs for databases, and consensus operations for private blockchain networks.
1. A tangible, non-transitory, machine-readable medium storing instructions that when executed by one or more processors effectuate operations comprising:
receiving a request for querying data within a database arrangement indicating at least a first database having a first data structure and a second database having a second data structure different from the first data structure, wherein:
the first database is a relational database,
the second database is a distributed database,
the first data structure comprises entries that correspond to rows of a table and data fields that correspond to columns of the table,
at least some of the data fields include references indicative of locations of respective plain text values stored within the second database, and
receiving a request comprises registering to obtain the request from an application configured to query data and receive data in a schema compatible with the first database and not the second database;
inspecting the request to identify that instructions for querying data in the first database include a query-language operator in the schema compatible with the first database by which the first database is requested to be searched with criteria satisfied by plain text values;
in response to identifying a query-language operator, determining that the identified query-language operator indicates a data field associated with a set of entries in the first database populated with respective references in a set of references, the set of references indicative of locations of respective plain text values stored within the second database that correspond to respective entries in the set of entries within the first database;
translating the query-language operator from the schema configured to identify plain text values satisfying the criteria within the data field into translated instructions for querying within the set of references in the first database, the translated instructions comprising a reference or subset of references that correspond to a location or locations of respective ones of the plain text values in the second database that satisfy the criteria of the query-language operator; and
submitting, based on the translated instructions, a query to the first database to retrieve an entry or a set of entries having the reference or a reference in the subset of references in the data field and providing a response to the request in the schema compatible with the first database.
2. The tangible, non-transitory, machine-readable medium of claim 1 , wherein retrieving an entry within the set of entries comprises:
obtaining a respective reference from the data field of the entry within the first database; and
requesting, based on the respective reference, the respective plain text value from the second database.
3. The tangible, non-transitory, machine-readable medium of claim 1 , wherein:
a respective reference stored for the data field of a respective entry is indicative of a location of a record within the second database that includes a respective plain text value, at least some respective references stored for the data field of respective ones of the entries are indicative of different locations within the second database, and
the different locations within the second database correspond to locations of different, distinct, records within the second database.
4. The tangible, non-transitory, machine-readable medium of claim 1 , wherein:
the set of references indicative of locations of respective plain text values stored within the second database comprises a different, distinct reference for each different, distinct location, and
the set of references includes, for at least two plain text values having a same value, different, distinct references.
5. The tangible, non-transitory, machine-readable medium of claim 1 , wherein:
the second data structure comprises a directed acyclic graph of cryptographic hash pointers.
6. The tangible, non-transitory, machine-readable medium of claim 5 , wherein:
a reference indicative of a location of a respective plain text values includes a cryptographic hash pointer or a transaction identifier corresponding to a node within the graph that contains at least a first portion of the respective plain text value.
7. The tangible, non-transitory, machine-readable medium of claim 1 , wherein:
the first data structure is a table in a relational database; and
the second data structure is a graph in a blockchain-based distributed database.
8. The tangible, non-transitory, machine-readable medium of claim 1 , wherein receiving a request for querying data within a database arrangement including at least a first database having a first data structure and a second database having a second data structure different from the first data structure comprises:
receiving the request from a client device, the request associated with the application configured to query data and receive data in the schema compatible with the first database and not the second database.
9. The tangible, non-transitory, machine-readable medium of claim 1 , wherein inspecting the request to identify whether instructions for querying data in the first database include a query-language operator in the schema compatible with the first database by which the first database is requested to be searched with criteria satisfied by plain text values comprises:
identifying an argument corresponding to the query-language operator; and
determining whether the argument specifies criteria that multiple potential plain text values satisfy.
10. The tangible, non-transitory, machine-readable medium of claim 1 , wherein inspecting the request to identify whether instructions for querying data in the first database include a query-language operator in the schema compatible with the first database by which the first database is requested to be searched with criteria satisfied by plain text values comprises:
identifying a statement of instructions for querying data;
evaluating the statement to identify one or more query-language operators of the statement;
determining, for the statement, by evaluating each query-language operator, which query-language operators include an argument specifying criteria that multiple potential plain text values satisfy.
11. The tangible, non-transitory, machine-readable medium of claim 10 , wherein:
the instructions for querying data are Structure Query Language instructions;
the statement includes two or more query-language operators; and
the two or more query-language operators are executed sequentially.
12. The tangible, non-transitory, machine-readable medium of claim 1 , wherein determining whether the identified query-language operator indicates a data field associated with a set of entries in the first database populated with respective references in a set of references comprises:
accessing a store including values indicative of which data fields within the first database include references replacing plain text values; and
determining whether an identifier of the data field is associated with an indication that the data field comprises references.
13. The tangible, non-transitory, machine-readable medium of claim 1 , wherein translating the query-language operator from the schema configured to identify plain text values satisfying the criteria within the data field into translated instructions for querying within the set of references in the first database comprises:
determining the reference or the subset of references that correspond to the location or the locations of the respective ones of the plain text values in the second database that satisfy the criteria of the query-language operator.
14. The tangible, non-transitory, machine-readable medium of claim 13 , wherein translating the operational clause from a schema configured to identify plain text values satisfying the criteria within the data field into translated instructions for querying within the set of references in the first database further comprises:
determining translated instructions comprising one or more of:
a new query-language operator comprising an argument including the reference or one or more of the references in the subset of references, or
for the query-language operator, a new argument including the reference or the subset of references; and
updating the instructions for querying data in the first database to include the new query-language operator or the new argument for the query-language operator.
15. The tangible, non-transitory, machine-readable medium of claim 1 , wherein translating the query-language operator from the schema configured to identify plain text values satisfying the criteria within the data field into translated instructions for querying within the set of references in the first database comprises:
querying a service based on an argument of the query-language operator to obtain the reference or the subset of references that correspond to the location or the locations of the respective ones of the plain text values in the second database that satisfy the criteria of the query-language operator;
determining translated instructions comprising one or more of:
a new query-language operator comprising an argument including the reference or one or more of the references in the subset of references, or
for the query-language operator, a new argument including the reference or the subset of references; and
updating the instructions for querying data in the first database to include the new query-language operator or the new argument for the query-language operator.
16. The tangible, non-transitory, machine-readable medium of claim 1 , wherein translating the query-language operator from the schema configured to identify plain text values satisfying the criteria within the data field into translated instructions for querying within the set of references in the first database comprises:
querying a service based on the query-language operator to obtain translated instructions for the query-language operator comprising one or more of:
a new query-language operator comprising an argument including the reference or one or more of the references in the subset of references, or
for the query-language operator, a new argument including the reference or the subset of references; and
updating the instructions for querying data in the first database to include the new query-language operator or the new argument for the query-language operator.
17. The tangible, non-transitory, machine-readable medium of claim 1 , wherein translating the query-language operator from the schema configured to identify plain text values satisfying the criteria within the data field into translated instructions for querying within the set of references in the first database comprises:
querying a service based on the request to obtain a translated request comprising translated instructions for the query-language operator comprising one or more of:
a new query-language operator comprising an argument including the reference or one or more of the references in the subset of references, or
for the query-language operator, a new argument including the reference or the subset of references.
18. The tangible, non-transitory, machine-readable medium of claim 1 , wherein submitting a query to the first database to retrieve an entry or a set of entries having the reference or a reference in the subset of references in the data field comprises:
executing a query comprising the translated instructions to retrieve the entry or the set of entries.
19. The tangible, non-transitory, machine-readable medium of claim 1 , wherein submitting a query to the first database to retrieve an entry or a set of entries having the reference or a reference in the subset of references in the data field comprises:
transmitting a query comprising the translated instructions to retrieve the entry or the set of entries.
20. A computer-implemented method comprising:
receiving, with a computing system, a request for querying data within a database arrangement indicating at least a first database having a first data structure and a second database having a second data structure different from the first data structure, wherein:
the first database is a relational database,
the second database is a distributed database,
the first data structure comprises entries that correspond to rows of a table and data fields that correspond to columns of the table,
at least some of the data fields include references indicative of locations of respective plain text values stored within the second database, and
receiving a request comprises registering to obtain the request from an application configured to query data and receive data in a schema compatible with the first database and not the second database;
inspecting, with the computing system, the request to identify that instructions for querying data in the first database include a query-language operator in the schema compatible with the first database by which the first database is requested to be searched with criteria satisfied by plain text values;
in response to identifying a query-language operator, determining, with the computing system, that the identified query-language operator indicates a data field associated with a set of entries in the first database populated with respective references in a set of references, the set of references indicative of locations of respective plain text values stored within the second database that correspond to respective entries in the set of entries within the first database;
translating, with the computing system, the query-language operator from the schema configured to identify plain text values satisfying the criteria within the data field into translated instructions for querying within the set of references in the first database, the translated instructions comprising a reference or subset of references that correspond to a location or locations of respective ones of the plain text values in the second database that satisfy the criteria of the query-language operator; and
submitting, with the computing system, based on the translated instructions, a query to the first database to retrieve an entry or a set of entries having the reference or a reference in the subset of references in the data field and providing a response to the request in the schema compatible with the first database.