Data set semantic similarity clustering
Embodiments of the system in include a method of clustering data sets based on the data set characteristics. A data repository stores a plurality of heterogeneous data sets. For a selected group of data sets (i) obtaining metadata; (ii) identifying upstream data sets; and (iii) extracting semantic features. The method further comprises clustering the plurality of data sets into one or more clusters based on similarity, wherein the similarity is determined based on a combination of the metadata, the upstream data sets, and the semantic features of the selected data sets. In one embodiment, the data sets are ranked within each cluster based on data set quality. The method enables the presenting of at least one cluster including one or more ranked data sets in response to a search query.
1 . A method comprising:
maintaining a data repository including a plurality of data sets, each data set comprising data from a data source with shared metrics;
processing a selected group of data sets of the plurality of data sets in the data repository by (i) obtaining metadata from the selected data sets; (ii) identifying upstream data sets utilized by the selected data sets; and (iii) extracting semantic features from the selected data sets;
clustering the plurality of data sets into one or more clusters based on similarity, wherein the similarity is determined based on a combination of the metadata, the upstream data sets, and the semantic features of the selected data sets;
ranking the data sets in a first cluster based on data set quality, wherein, for a data set, a data set quality ranking comprises a metadata quality rating computed using a first quality function and a data content quality rating computed using a second quality function;
storing the first cluster and ranking data for each of the selected group of data sets in a data set cluster repository; and
initiating a data set merging process to remove the data set when a combination of a metadata match and a semantic match are above a similarity threshold, wherein the data set merging process is performed based on the data set quality ranking.
2 . The method of claim 1 further comprising:
evaluating data set quality for the selected group of data sets, the data set quality comprising a metadata quality calculated from one or more of: arrival time, update frequency, number of columns, or raw data count.
3 . The method of claim 1 , further comprising:
identifying two data sets in the first cluster, the two data sets having identical upstream data sets;
comparing the metadata for the two data sets;
determining a percentage metadata match;
comparing the semantic features for the two data sets;
determining a percentage semantic match; and
initiating the data set merging process when a combination of the percentage metadata match and the percentage semantic match are above a similarity threshold.
4 . The method of claim 3 , further comprising:
wherein a merged data set comprises a single data set including fields and data content of the two data sets that were merged.
5 . The method of claim 3 , further comprising:
comparing a list of field labels in the two data sets;
identifying a field having matching content and different field labels; and
verifying identity between the field labels by comparing data set entries in the field.
6 . The method of claim 1 , wherein the metadata for a data set comprises one or more of: a number of attributes, a list of attribute names, field types, or value constraints.
7 . The method of claim 1 , wherein extracting the semantic features comprises extracting a representative subset of data set content.
8 . The method of claim 1 , further comprising:
featurizing the metadata, the upstream data sets, and the semantic features for use by the clustering.
9 . The method of claim 1 , wherein the clustering utilizes K-means clustering based on a combination of a plurality of the metadata, the upstream data sets, and the semantic features.
10 . A method comprising:
maintaining a data repository including a plurality of data sets;
selecting a group of data sets for clustering, each data set comprising data from a data source with shared metrics;
for each of the data sets in the group of data sets: (i) obtaining metadata comprising number of attributes, a list of attribute names, field types, and value constraints, (ii) iteratively identifying upstream data sets utilized by the data set, and (iii) extracting semantic features;
clustering the group of data sets into one or more clusters based on data set similarity, the data set similarity determined based on a combination of the metadata, the semantic features, and the upstream data sets;
ranking the data sets in each cluster based on data set quality, wherein, for a data set, a data set quality ranking comprises a metadata quality rating computed using a first quality function and a data content quality rating computed using a second quality function;
storing the ranking and the clustering in a data set cluster repository;
initiating a data set merging process to remove the data set when a combination of a metadata match and a semantic match are above a similarity threshold, wherein the data set merging process is performed based on the data set quality ranking;
receiving a search query for a data set;
retrieving a first cluster responsive to the search query from the data set cluster repository; and
responding to the search query with the first cluster, in ranked order.
11 . A computer implemented system comprising:
a data set info extractor implemented by a processor to (i) obtain metadata from a group of data sets; (ii) identify upstream data sets utilized by the group of data sets; and
(iii) extract semantic features from the group of data sets, wherein each data set comprises data from a data source with shared data metrics;
a data set clusterer to cluster the group of data sets into one or more clusters based on similarity, wherein the similarity is determined based on a combination of the metadata, the upstream data sets, and the semantic features of the group of data sets;
a data cluster quality ranker to rank the data sets in a first cluster based on data set quality, wherein, for a data set, a data set quality ranking comprises a metadata quality rating computed using a first quality function and a data content quality rating computed using a second quality function;
a data set cluster repository to store cluster data and rank data for each of the data sets;
a memory to store the cluster data and the rank data;
a data set merger system to initiate a data set merging process to remove the data set when a combination of a metadata match and a semantic match are above a similarity threshold, wherein the data set merging process is performed based on the data set quality ranking; and
a user interface to receive a search query, the computer implemented system to identify a responsive cluster to the search query based on the cluster data and the rank data in memory and respond with the responsive cluster of ranked data sets.
12 . The system of claim 11 further comprising:
a data set quality monitor to evaluate data set quality for the group of data sets; and
a user interface to present the ranked data sets in the first cluster in response to a search query.
13 . The system of claim 11 , further comprising:
the data set merger system to identify two data sets in a cluster for merger, and the data set merger system to (i) compare the metadata for the two data sets and determine a percentage metadata match, (ii) compare the semantic features for the two data sets and determine a percentage semantic match, (iii) compare the upstream data sets for the two data sets and determine a percentage upstream data set match; and
the data set merger system to merge the two data sets when a combination of the metadata match and the semantic match are above a similarity threshold, to reduce memory storage requirements for the group of data sets.
14 . The system of claim 13 , further comprising wherein a merged data set comprises a single data set including fields and data content of the two data sets that were merged.
15 . The system of claim 13 , wherein the data set merger system is further configured to:
compare a list of field labels in the two data sets;
identify a field having matching content and different field labels; and
verify identity between the field labels by comparing data set entries in the field.
16 . The system of claim 11 , wherein the metadata for a data set comprises one or more of: a number of attributes, a list of attribute names, field types, or value constraints.
17 . The system of claim 11 , wherein extracting semantic features includes a representative subset of data set content.
18 . The system of claim 11 , further comprising:
featurizing the metadata, the upstream data sets, and the semantic features for use by the data set clusterer.
19 . The system of claim 11 , wherein the data set clusterer utilizes K-means clustering based on a combination of a plurality of the metadata, the upstream data sets, and the semantic features.
20 . The system of claim 11 , further comprising:
the data cluster quality ranker further to calculate the data set quality by assigning a rating to each quality element and multiplying the rating by a weight assigned to the quality element.
21 . The method of claim 1 , further comprising:
receiving a search query, the search query requesting one or more responsive data sets from the data repository;
identifying the first cluster responsive to the search query in the data set cluster repository; and
responding to the search query with the ranked data set in the first cluster.