Dataset networking and database modeling
In various example embodiments, a system and methods are presented for generating node clusters from a plurality of disconnected nodes and generating data access models for interaction with the nodes. The system and methods identify one or more datasets associated with a first set of nodes distributed across a plurality of node clusters, and a set of entities associated within the plurality of node clusters. A node layer is generated based on the one or more datasets and the set of entities. One or more connections are generated between the first set of nodes and a set of coordinating nodes, and between the set of coordinating nodes and a second set of nodes. The systems and methods generate a result set distributed across the plurality of nodes based on connections between the set of coordinating nodes and the first set of nodes and the second set of nodes.
1. A method for analyzing data in a plurality of datasets, including a first dataset and a second dataset, the first dataset and the second dataset involving heterogeneous health datasets, the method comprising:
transforming, by one or more hardware processors, the first dataset into a first plurality of isolated node clusters, each node cluster comprising a plurality of nodes connected by edges;
identifying, by the one or more hardware processors, a set of entities associated with the first plurality of isolated node clusters;
generating, by the one or more hardware processors, a node layer comprising a set of coordinating nodes based at least in part on the set of entities, wherein the one or more hardware processors selects the coordinating nodes based on a confidence that the selected coordinating node is a canonical representation of one or more entities in the set of entities;
generating, by the one or more hardware processors, one or more connections between nodes of the first plurality of isolated node clusters and the set of coordinating nodes;
transforming, by one or more hardware processors, the second dataset into a second plurality of isolated node clusters, each node cluster comprising a plurality of nodes connected by edges;
generating, by the one or more hardware processors, one or more connections between nodes of the second plurality of isolated node clusters and the set of coordinating nodes; and
responsive to a query, generating, by the one or more hardware processors, a result set distributed across the first dataset and the second dataset based on at least a portion of the connections between the set of coordinating nodes and one or more nodes of the first plurality of isolated node clusters and the second plurality of isolated node clusters.
2. The method of claim 1 , comprising:
generating a data access model for interconnecting the first plurality of isolated node clusters and the second plurality of isolated node clusters.
3. The method of claim 2 , wherein generating the result set comprises:
selectively removing edges connecting a first portion of nodes of the second plurality of isolated node clusters and the set of coordinating nodes;
verifying the edges within the data access model exclude edges connecting the first portion of nodes of the second plurality of isolated node clusters; and
generating the result set, responsive to the query, distributed across the first dataset and the second dataset.
4. The method of claim 3 , comprising:
identifying presentation of the result set to a computing device associated with the query; and
based on presentation of the result set, reestablishing the edges connecting the first portion of nodes of the second plurality of isolated node clusters and the set of coordinating nodes.
5. The method of claim 3 , wherein selectively removing the edges of the first portion of nodes comprises:
identifying an access permission within the query, the access permission corresponding to nodes of the second plurality of node clusters available to the query;
identifying the first portion of nodes is excluded from the access permission; and
removing the edges of the first portion of nodes prior to execution of the query by the data access model.
6. The method of claim 1 , wherein generating the result set comprises:
identifying a field value within the query;
determining a candidate match of a field associated with a node of the second plurality of isolated node clusters; and
selecting a value for the candidate match of the node of the second plurality of isolated node clusters for inclusion in the result set.
7. The method of claim 1 , wherein generating the one or more connections between the nodes of the first plurality of isolated node clusters and the set of coordinating nodes comprises:
when a node of the first plurality of isolated node clusters comprises a first entity that is within a correction distance of a second entity represented by a coordinating node of the set of coordinating nodes, generating a connection between the node of the first plurality of isolated node clusters comprising the first entity and the coordinating node representing the second entity.
8. The method of claim 1 , wherein generating the node layer further comprises:
generating the set of coordinating nodes;
identifying features associated with edges connecting each coordinating node of the node layer with at least one node of the first plurality of isolated node clusters; and
for each coordinating node, populating features of the coordinating node with one or more features identified with a connected node of the first plurality of isolated node clusters.
9. The method for analyzing data in a plurality of datasets of claim 1 ,
wherein the plurality of datasets comprise clinical research datasets.
10. A system for analyzing data in a plurality of datasets, including a first dataset and a second dataset, the first dataset and the second dataset involving heterogeneous health datasets, comprising:
one or more processors; and
a processor-readable storage medium storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
transforming the first dataset into a first plurality of isolated node clusters, each node cluster comprising a plurality of nodes connected by edges;
identifying a set of entities associated with the first plurality of isolated node clusters;
generating a node layer comprising a set of coordinating nodes based at least in part on the set of entities, wherein the one or more hardware processors selects the coordinating nodes based on a confidence that the selected coordinating node is a canonical representation of one or more entities in the set of entities;
generating one or more connections between nodes of the first plurality of isolated node clusters and the set of coordinating nodes;
transforming the second dataset into a second plurality of isolated node clusters, each node cluster comprising a plurality of nodes connected by edges;
generating, by the one or more hardware processors, one or more connections between nodes of the second plurality of isolated node clusters and the set of coordinating nodes; and
responsive to a query, generating a result set distributed across the first dataset and the second dataset based on at least a portion of the connections between the set of coordinating nodes and one or more nodes of the first plurality of isolated node clusters and the second plurality of isolated node clusters.
11. The system of claim 10 , comprising:
generating a data access model for interconnecting the first plurality of isolated node clusters and the second plurality of isolated node clusters.
12. The system of claim 11 , wherein generating the result set comprises:
selectively removing edges connecting a first portion of nodes of the second plurality of isolated node clusters and the set of coordinating nodes;
verifying the edges within the data access model exclude edges connecting the first portion of nodes of the second plurality of isolated node clusters; and
generating the result set, responsive to the query, distributed across the first dataset and the second dataset.
13. The system of claim 12 , comprising:
identifying presentation of the result set to a computing device associated with the query; and
based on presentation of the result set, reestablishing the edges connecting the first portion of nodes of the second plurality of isolated node clusters and the set of coordinating nodes.
14. The system of claim 12 , wherein selectively removing the edges of the first portion of nodes comprises:
identifying an access permission within the query, the access permission corresponding to nodes of the second plurality of node clusters available to the query;
identifying the first portion of nodes is excluded from the access permission; and
removing the edges of the first portion of nodes prior to execution of the query by the data access model.
15. The system of claim 10 , wherein generating the result set comprises:
identifying a field value within the query;
determining a candidate match of a field associated with a node of the second plurality of isolated node clusters; and
selecting a value for the candidate match of the node of the second plurality of isolated node clusters for inclusion in the result set.
16. The system of claim 10 , wherein generating the node layer further comprises:
generating the set of coordinating nodes;
identifying features associated with edges connecting each coordinating node of the node layer with at least one node of the first plurality of isolated node clusters; and
for each coordinating node, populating features of the coordinating node with one or more features identified with a connected node of the first plurality of isolated node clusters.
17. A non-transitory processor-readable storage medium comprising instructions for analyzing data in a plurality of datasets, including a first dataset and a second dataset, the first dataset and the second dataset involving heterogeneous health datasets, the instructions, when executed by one or more hardware processors of a machine, causing the machine to implement operations comprising:
transforming the first dataset into a first plurality of isolated node clusters, each node cluster comprising a plurality of nodes connected by edges;
identifying a set of entities associated with the first plurality of isolated node clusters;
generating a node layer comprising a set of coordinating nodes based at least in part on the set of entities, wherein the one or more hardware processors selects the coordinating nodes based on a confidence that the selected coordinating node is a canonical representation of one or more entities in the set of entities;
generating one or more connections between nodes of the first plurality of isolated node clusters and the set of coordinating nodes;
transforming the second dataset into a second plurality of isolated node clusters, each node cluster comprising a plurality of nodes connected by edges;
generating, by the one or more hardware processors, one or more connections between nodes of the second plurality of isolated node clusters and the set of coordinating nodes; and
responsive to a query, generating a result set distributed across the first dataset and the second dataset based on at least a portion of the connections between the set of coordinating nodes and one or more nodes of the first plurality of isolated node clusters and the second plurality of isolated node clusters.
18. The processor-readable storage medium of claim 17 , comprising:
generating a data access model for interconnecting the first plurality of isolated node clusters and the second plurality of isolated node clusters.
19. The processor-readable storage medium of claim 18 , wherein generating the result set comprises:
selectively removing edges connecting a first portion of nodes of the second plurality of isolated node clusters and the set of coordinating nodes;
verifying the edges within the data access model exclude edges connecting the first portion of nodes of the second plurality of isolated node clusters; and
generating the result set, responsive to the query, distributed across the first dataset and the second dataset.
20. The processor-readable storage medium of claim 19 , comprising:
identifying presentation of the result set to a computing device associated with the query; and
based on presentation of the result set, reestablishing the edges connecting the first portion of nodes of the second plurality of isolated node clusters and the set of coordinating nodes.
21. The processor-readable storage medium of claim 19 , wherein selectively removing the edges of the first portion of nodes comprises:
identifying an access permission within the query, the access permission corresponding to nodes of the second plurality of node clusters available to the query;
identifying the first portion of nodes is excluded from the access permission; and
removing the edges of the first portion of nodes prior to execution of the query by the data access model.