Platform management of integrated access of public and privately-accessible datasets utilizing federated query generation and query schema rewriting optimization
Various techniques are described for platform management of integrated access of public and privately-accessible datasets utilizing federated query generation and query schema rewriting optimization, including receiving at a dataset access platform a query formatted according to a first data schema, generating a copy of the query, saving the query and the copy to a datastore, parsing the copy of the query in the first schema using an inference engine, determining whether the query comprises data associated with an access control condition associated with accessing the dataset, the access control condition being configured to indicate whether the query is permitted to access the dataset, and rewriting, using a proxy server, the copy of the query in a second schema, and optimizing the rewriting by identifying a database engine to execute the query and including other data converted into another triple associated with an attribute of the query.
1. A method, comprising:
receiving a query at a dataset access platform, the query being formatted according to a first data schema, the query comprising data associated with a request to access a dataset;
generating a copy of the query;
identifying whether the query is a master or a replica as the copy of the query;
identifying a datastore for storing the query as either the master or the copy, or both;
updating a graph as a data model associated with the query to identify elements to distinguish the copy for data operations to be performed;
parsing the copy of the query in the first schema, the parsing being performed by an inference engine configured to identify the dataset, to infer an attribute associated with the query, and to generate one or more data links between the dataset and another dataset accessible by the dataset access platform, wherein parsing the copy of the query further includes parsing the query into a data structure including an abstract syntax tree associated with a target query language;
rewriting the copy of the query in a second schema including a triples-based format and, if the attribute indicates the query is configured to provide authentication data to access the dataset, the rewriting comprising converting the copy of the query into a triple and converting the attribute into another triple;
optimizing rewriting the copy of the query,
determining one or more property paths to the another dataset;
identifying a database engine to execute the query, the database engine is configured to be topologically internal to a data network associated with the dataset access platform; and
converting other data to a further triple, the other data and the further triple being associated with a path configured to route the query or the copy of the query in the second schema including the triples-based format from the dataset access platform to retrieve query results from a target database configured to store the dataset as graph-based data,
wherein the one or more property paths are determined by performing a comparison of another attribute associated with each of the one or more property paths to a threshold to identify an optimal path to run the query or the copy of the query.
2. The method of claim 1 , wherein the first data schema is configured to provide data in a relational-based format.
3. The method of claim 1 , wherein the copy is identified as the master.
4. The method of claim 1 , wherein the converting the copy and the converting the attribute are performed using a framework.
5. The method of claim 1 , further comprising determining whether the attribute comprises an access control condition associated with accessing the dataset.
6. The method of claim 5 , wherein the access control condition is configured to authenticate the query to permit access to the dataset.
7. The method of claim 1 , wherein the inference engine is configured to generate the graph associated with the dataset, the another dataset, the one or more data links, and one or more property paths, the graph being stored in the datastore or another datastore.
8. The method of claim 1 , wherein the triple and the another triple are configured to be included in a rewritten query generated from the rewriting, the rewritten query being configured to be processed by one or more endpoints associated with the dataset access platform.
9. The method of claim 8 , wherein at least one of the one or more endpoints comprise a database engine, the database engine being configured to execute the query, the copy of the query, or the rewritten query using a framework to perform data communication with the dataset or another data network topologically associated with the dataset.
10. The method of claim 1 , further comprising:
identifying a database engine to execute the query, the database engine being configured to communicate data between the dataset access platform and another data network associated with the dataset.
11. The method of claim 1 , further comprising:
identifying a database engine configured to use the triple, the another triple, and a further triple to execute the copy of the query to access the dataset and, after accessing the dataset, returning responsive data to the dataset access platform.
12. The method of claim 1 , wherein the first schema includes a structured relational-based format includes SQL query data.
13. A system comprising:
a memory including executable instructions; and
a processor configured to execute the instructions to:
receive a query at a dataset access platform, the query being formatted according to a first data schema, the query comprising data associated with a request to access a dataset;
generate a copy of the query;
identify whether the query is a master or a replica as the copy of the query;
identify a datastore for storing the query as either the master or the copy, or both;
update a graph as a data model associated with the query to identify elements to distinguish the copy for data operations to be performed;
parse the copy of the query in the first schema, the parsing being performed by an inference engine configured to identify the dataset, to infer an attribute associated with the query, and to generate one or more data links between the dataset and another dataset accessible by the dataset access platform;
rewrite the copy of the query in a second schema including a triples-based format and, if the attribute indicates the query is configured to provide authentication data to access the dataset, the rewriting comprising converting the copy of the query into a triple and converting the attribute into another triple;
optimize rewriting the copy of the query including:
identifying a database engine to execute the query; and
converting other data to a further triple, the other data and the further triple being associated with a path configured to route the query or the copy of the query in the second schema including the triples-based format from the dataset access platform to retrieve query results from a target database configured to store the dataset as graph-based data.
14. The system of claim 13 , wherein the first data schema is configured to provide data in a relational-based format.