Systems and methods for 3D data visualization and network extraction
Systems and methods for data visualization and network extraction in accordance with embodiments of the invention are illustrated. One embodiment includes a method including obtaining a graph comprising a plurality of nodes and a plurality of edges, identifying a plurality of communities in the graph, where each community includes nodes from the plurality of nodes, generating a community graph structure based on the identified communities, where the community graph includes a plurality of supernodes and a plurality of superedges, spatializing the community graph structure, unpacking the spatialized community graph structure into an unpacked graph structure comprising the plurality of nodes and the plurality of edges, where each node in the plurality of nodes is located at approximately the position of the supernode that represented it, spatializing the unpacked graph structure, and providing the spatialized unpacked graph structure.
1 . A system for spatializing data in a virtual space, comprising:
a processor; and
a memory containing a data visualization application that directs the processor to:
obtain a network graph comprising of a first plurality of nodes and a first plurality of edges;
identify communities of nodes in the network graph structure;
label each node in the first plurality of nodes with the community in which said node is a member;
generate a community graph representation of the network graph, where the community graph representation comprises:
a second plurality of nodes; and
a second plurality of edges;
where each node in the second plurality of nodes represents nodes in the first plurality of nodes having the same community label; and
where each given edge in the second plurality of edges connects two nodes in the second plurality of nodes, and represents edges in the first plurality of edges that connect pairs of nodes in the first plurality of nodes having community labels equal to the two community labels associated with the nodes connected by the given edge;
spatialize the community graph representation;
unpack the spatialized community graph representation into an intermediate network graph structure, where the intermediate network graph structure comprises the first plurality of nodes and the first plurality of edges, and where the locations of the first plurality of nodes in the intermediate network graph structure approximate the location in the community graph representation of the node representing the same community label in the second plurality of nodes;
spatialize the intermediate network graph structure;
provide the spatialized intermediate network graph structure as a spatialized network graph;
generate a list of all node pairs in the first plurality of nodes;
randomly subsample the list;
calculate centrality for each of the subsampled node pairs using Dijkstra's algorithm;
identify the most central node from the subsampled nodes:
repeat the random subsampling, centrality calculation, and identification of the most central node until the most central node does not change between iterations to identify the most central community in the spatialized network.
2 . The system of claim 1 , wherein to identify communities, the data visualization application directs the processor to apply Markov Clustering to nodes in the first plurality of nodes to generate community labels.
3 . The system of claim 2 , wherein to identify communities, the data visualization application further directs the processor to refine the generated community labels using Louvain Modularity.
4 . The system of claim 2 , wherein the processor comprises at least one graphics processing unit, wherein a pruning threshold for the Markov Clustering is the number of nodes in the first plurality of nodes.
5 . The system of claim 1 , wherein to provide the spatialized network graph, the data visualization application directs the processor to display the spatialized network graph in a virtual 3D environment via a display device.
6 . The system of claim 1 , wherein to obtain the network graph, the data visualization application further directs the processor to extract the network graph from a tabular database.
7 . The system of claim 1 , wherein the data visualization application further directs the processor to identify a community leader node for each community in the spatialized network graph.
8 . The system of claim 1 , wherein to identify the most central community, the data visualization application directs the processor to:
randomly subsample nodes in the first plurality of nodes without replacement until a predetermined threshold is reached;
for each subsampled node, calculate the graph distance between the subsampled node and all nodes in the first plurality of nodes in the spatialized network graph; and
approximate centrality based on the calculated distances.
9 . A method for spatializing data in a virtual space, comprising:
obtaining a network graph comprising of a first plurality of nodes and a first plurality of edges;
identifying communities of nodes in the network graph structure;
labeling each node in the first plurality of nodes with the community in which said node is a member;
generating a community graph representation of the network graph, where the community graph representation comprises:
a second plurality of nodes; and
a second plurality of edges;
where each node in the second plurality of nodes represents nodes in the first plurality of nodes having the same community label; and
where each given edge in the second plurality of edges connects two nodes in the second plurality of nodes, and represents edges in the first plurality of edges that connect pairs of nodes in the first plurality of nodes having community labels equal to the two community labels associated with the nodes connected by the given edge;
spatializing the community graph representation;
unpacking the spatialized community graph representation into an intermediate network graph structure, where the intermediate network graph structure comprises the first plurality of nodes and the first plurality of edges, and where the locations of the first plurality of nodes in the intermediate network graph structure approximate the location in the community graph representation of the node representing the same community label in the second plurality of nodes;
spatializing the intermediate network graph structure;
providing the spatialized intermediate network graph structure as a spatialized network graph; and
identifying the most central community in the spatialized network graph by:
generating a list of all node pairs in the first plurality of nodes;
randomly subsampling the list;
calculating centrality for each of the subsampled node pairs using Dijkstra's algorithm;
identifying the most central node from the subsampled nodes;
repeating the random subsampling, centrality calculation, and identification of the most central node until the most central node does not change between iterations.
10 . The method of claim 9 , wherein identifying communities comprises applying Markov Clustering to nodes in the first plurality of nodes to generate community labels.
11 . The method of claim 10 , wherein identifying communities comprises refining the generated community labels using Louvain Modularity.
12 . The method of claim 10 , wherein the processor comprises at least one graphics processing unit, wherein a pruning threshold for the Markov Clustering is the number of nodes in the first plurality of nodes.
13 . The method of claim 9 , wherein providing the spatialized network graph comprises displaying the spatialized network graph in a virtual 3D environment via a display device.
14 . The method of claim 9 , wherein obtaining the network graph comprises extracting the network graph from a tabular database.
15 . The method of claim 9 , further comprising identifying a community leader node for each community in the spatialized network graph.
16 . The method of claim 9 , wherein identifying the most central community comprises:
randomly subsampling nodes in the first plurality of nodes without replacement until a predetermined threshold is reached;
for each subsampled node, calculating the graph distance between the subsampled node and all nodes in the first plurality of nodes in the spatialized network graph; and
approximating centrality based on the calculated distances.