Methods and systems for extracting and visualizing patterns in large-scale data sets
Disclosed are systems and methods for extracting and visualizing patterns in large-scale data sets. A data set comprising a plurality of data points is received. A plurality of connections between the plurality of data points is generated. Based on the plurality of connections, a plurality of groups from the plurality of data points is generated. A visual analytic comprising a plurality of geometrical shapes corresponding to the plurality of groups is generated.
1. A method comprising:
generating, by a computing device, based on a plurality of data points of an in-memory data set, a tree structure connecting the plurality of data points and comprising a plurality of edges, each edge having a length;
removing, from the tree structure, at least one edge of the plurality of edges, wherein the length of the at least one edge is greater than or equal to a sum of an average of the lengths of the plurality of edges plus a standard deviation of the lengths of the plurality of edges;
determining, based on a plurality of clusters resulting from the removal of the at least one edge from the tree structure, an inter-group similarity measurement and an intra-group similarity measurement for each cluster of the plurality of clusters, wherein each cluster of the plurality of clusters comprises a respective subset of the plurality of data points;
generating, based on the inter-group similarity measurement and the intra-group similarity measurement for two or more clusters of the plurality of clusters, at least one merged cluster comprising the respective subsets of the plurality of data points associated with the two or more clusters, wherein the at least one merged cluster comprises a convex hull, and wherein the convex hull comprises an internal structure defined by a plurality of hull edges surrounding the respective subsets of the plurality of data points associated with the two or more clusters;
causing, based on the internal structure of the convex hull, a subset of the plurality of hull edges to be split, resulting in an optimized convex hull, wherein each hull edge, of the subset of the plurality of hull edges, does not cut into the internal structure of the optimized convex hull after being split; and
rendering, at a user interface of the computing device, a visual representation of the at least one merged cluster with the optimized convex hull, wherein the visual representation is rendered such that the visual representation does not depict the respective subsets of the plurality of data points associated with the two or more clusters, and wherein the user interface facilitates querying the in-memory data set via interaction with the visual representation.
2. The method of claim 1 , wherein generating the tree structure comprises at least one of:
generating, based on a Delaunay triangulation applied to the plurality of data points, a minimum spanning tree (MST);
generating, based on at least one common variable associated with each of the plurality of data points, a plurality of connections for an MST; or
generating, based on at least one distance metric, an MST, wherein the at least one distance metric comprises a Euclidean distance or a Manhattan distance.
3. The method of claim 1 , wherein each cluster of the two or more clusters comprises a subset of the plurality of data points.
4. The method of claim 3 , wherein the intra-group similarity measurement for each of the two or more clusters is indicative of a level of similarity between data points within the subset.
5. The method of claim 3 , wherein the inter-group similarity measurement for each of the two or more clusters is indicative of a level of similarity between that cluster and the remaining clusters of the plurality of clusters.
6. The method of claim 1 , further comprising: determining, based on an absence of a predefined number of clusters associated with the removal of the at least one edge from the tree structure, that a supervised classification is to be applied to the plurality of clusters.
7. The method of claim 1 , further comprising: determining, based on an absence of a maximum number of clusters associated with the removal of the at least one edge from the tree structure, that a supervised classification is to be applied to the plurality of clusters.
8. A method comprising:
generating, by a computing device, based on a plurality of data points of an in-memory data set, a tree structure connecting the plurality of data points and comprising a plurality of edges, each edge having a length;
removing, from the tree structure, at least one edge of the plurality of edges, wherein the length of the at least one edge is greater than or equal to a sum of an average of the lengths of the plurality of edges plus a standard deviation of the lengths of the plurality of edges;
determining that a plurality of clusters resulting from the removal of the at least one edge from the tree structure is associated with a defined quantity, wherein each cluster of the plurality of clusters comprises a respective subset of the plurality of data points, and wherein the defined quantity of the plurality of clusters comprises a convex hull, wherein the convex hull comprises an internal structure defined by a plurality of hull edges surrounding the respective subsets of the plurality of data points;
causing, for each cluster of the plurality of clusters, based on the internal structure of the convex hull, a subset of the plurality of hull edges to be split, resulting in an optimized convex hull, wherein each hull edge, of the subset of the plurality of hull edges, does not cut into the internal structure of the optimized convex hull after being split; and
rendering, at a user interface, a visual representation of the plurality of clusters with the optimized convex hulls, wherein the visual representation does not depict the respective subsets of the plurality of data points associated with the plurality of clusters, and wherein the user interface facilitates querying the in-memory data set via interaction with the visual representation.
9. The method of claim 8 , wherein generating the tree structure comprises generating, based on a Delaunay triangulation applied to the plurality of data points, a minimum spanning tree.
10. The method of claim 8 , wherein generating the tree structure comprises:
generating, based on at least one common variable associated with each of the plurality of data points, a plurality of connections for a minimum spanning tree (MST); and
generating, based on the plurality of connections, the MST.
11. The method of claim 8 , wherein generating the tree structure comprises generating, based on at least one distance metric, a minimum spanning tree.
12. The method of claim 11 , wherein the at least one distance metric comprises a Euclidean distance or a Manhattan distance.
13. The method of claim 8 , wherein the defined quantity of the plurality of clusters comprises a maximum quantity of clusters.
14. The method of claim 8 , wherein the defined quantity of the plurality of clusters comprises a predetermined quantity of clusters.
15. An apparatus comprising:
one or more processors, and
memory storing processor executable instructions that, when executed by the one or more processors, cause the apparatus to:
generate, based on a plurality of data points of an in-memory data set, a tree structure connecting the plurality of data points and comprising a plurality of edges, each edge having a length;
remove, from the tree structure, at least one edge of the plurality of edges, wherein the length of the at least one edge is greater than or equal to a sum of an average of the lengths of the plurality of edges plus a standard deviation of the lengths of the plurality of edges;
determine, based on a plurality of clusters resulting from the removal of the at least one edge from the tree structure, an inter-group similarity measurement and an intra- group similarity measurement for each cluster of the plurality of clusters, wherein each cluster of the plurality of clusters comprises a respective subset of the plurality of data points;
generate, based on the inter-group similarity measurement and the intra-group similarity measurement for two or more clusters of the plurality of clusters, at least one merged cluster comprising the respective subsets of the plurality of data points associated with the two or more clusters, wherein the at least one merged cluster comprises a convex hull, and wherein the convex hull comprises an internal structure defined by a plurality of hull edges surrounding the respective subsets of the plurality of data points associated with the two or more clusters;
cause, based on the internal structure of the convex hull, a subset of the plurality of hull edges to be split, resulting in an optimized convex hull, wherein each hull edge, of the subset of the plurality of hull edges, does not cut into the internal structure of the optimized convex hull after being split; and
render, at a user interface, a visual representation of the at least one merged cluster with the optimized convex hull, wherein the visual representation is rendered such that the visual representation does not depict the respective subsets of the plurality of data points associated with the two or more clusters, and wherein the user interface facilitates querying the in-memory data set via interaction with the visual representation.
16. The apparatus of claim 15 , wherein the processor executable instructions that cause the apparatus to generate the tree structure further cause the apparatus to at least one of:
generate, based on a Delaunay triangulation applied to the plurality of data points, the a minimum spanning tree (MST);
generate, based on at least one common variable associated with each of the plurality of data points, a plurality of connections for an MST; or
generate, based on at least one distance metric, an MST, wherein the at least one distance metric comprises a Euclidean distance or a Manhattan distance.
17. The apparatus of claim 15 , wherein each cluster of the two or more clusters comprises a subset of the plurality of data points.
18. The apparatus of claim 17 , wherein the intra-group similarity measurement for each of the two or more clusters is indicative of a level of similarity between data points within that cluster and the remaining clusters of the plurality of clusters.
19. The apparatus of claim 18 , wherein the inter-group similarity measurement is indicative of a level of similarity between each subset.
20. The apparatus of claim 15 , wherein the processor executable instructions further cause the apparatus to determine, based on an absence of a predefined number of clusters associated with a result of a coarse classification, or based on an absence of a maximum number of clusters associated with the removal of the at least one edge from the tree structure, that a supervised classification is to be applied to the plurality of clusters.