Machine learning techniques for predictive schema analysis
Various embodiments of the present invention provide methods, apparatus, systems, computing devices, computing entities, and/or the like for performing predictive structural analysis. Certain embodiments of the present invention utilize systems, methods, and computer program products that perform predictive structural analysis using at least one of table column classification machine learning models, table column clustering machine learning models, structural variance generation machine learning models, and emergence report generation machine learning models.
1 . A computer-implemented method comprising:
generating, by one or more processors and for a table column of a selected dataset, among a plurality of datasets, associated with a null overall column type, (i) a plurality of predicted column types comprising a predicted header-based column type, a predicted data-based column type, a predicted entity-recognition-based column type, and a predicted pattern-matching-based column type, and (ii) a plurality of column type voting weights comprising a header-based column type voting weight, a data-based column type voting weight, an entity-recognition-based column type voting weight, and a pattern-matching-based column type voting weight, by:
generating, via a header-based table classification machine learning model that is (i) trained to identify columns based at least in part on column headers and (ii) characterized by a first set of decision tree data objects, and based at least in part on a table column name set for the table column, the predicted header-based column type for the table column and the header-based column type voting weight for the predicted header-based column type, wherein the header-based column type voting weight is a data value indicating a first inferred credibility score based at least in part on a subset of the first set of decision tree data objects which have selected the predicted header-based column type,
generating, via a data-based table classification machine learning model that is (i) trained to identify columns based at least in part on column values and (ii) characterized by a second set of decision tree data objects, and based at least in part on a table column value set for the table column, the predicted data-based column type for the table column and the data-based column type voting weight for the predicted data-based column type, wherein the data-based column type voting weight is a data value indicating a second inferred credibility score based at least in part on a subset of the second set of decision tree data objects which have selected the predicted data-based column type;
generating, via an entity recognition classification machine learning model and based at least in part on the table column value set, the predicted entity-recognition-based column type for the table column and the entity-recognition-based column type voting weight for the predicted entity-recognition-based column type, and
generating, via a pattern matching classification machine learning model and based at least in part on the table column value set, the predicted pattern-matching-based column type for the table column and the pattern-matching-based column type voting weight for the predicted pattern-matching-based column type;
generating, by the one or more processors and via a voting machine learning model and based at least in part on the plurality of predicted column types and the plurality of column type voting weights, an overall column type prediction for the table column;
generating, by the one or more processors and via a structural variance generation machine learning model and based at least in part on a comparison between the table column and at least one reference table column from at least one additional dataset from the plurality of datasets, a table column pair similarity measure;
generating, by the one or more processors, responsive to a determination that the table column pair similarity measure exceeds a threshold, an unidentified table column grouping comprising the table column and the at least one reference table column;
generating, by the one or more processors, an emergence report indicating updated data corresponding to the table column of the selected dataset, wherein the updated data comprises (i) the unidentified table column grouping and (ii) the overall column type prediction for the table column; and
providing, by the one or more processors, access to the updated data to one or more users by generating a prediction output user interface configured to display the emergence report via one or more external computing entities operable by the one or more users.
2 . The computer-implemented method of claim 1 , wherein generating the overall column type prediction for a table column comprises:
for a candidate column type of a plurality of candidate column types:
identifying a predicted column type set for the table column that corresponds to the candidate column type,
identifying a column type voting weight set that corresponds to the predicted column type set, and
determining a candidate column type voting value for the candidate column type with respect to the table column based at least in part on the column type voting weight set; and
generating the overall column type prediction based at least in part on the candidate column type voting value for the candidate column type with respect to the table column.
3 . The computer-implemented method of claim 2 , wherein generating the overall column type prediction based at least in part on the candidate column type voting value for a candidate column type with respect to the table column comprises:
determining a maximal candidate column type voting value that is associated with the table column,
determining whether the maximal candidate column type voting value satisfies a candidate column type voting value threshold, and
in response to determining that the maximal candidate column type voting value fails to satisfy the candidate column type voting value threshold, generating the overall column type prediction based at least in part on the candidate column type that is associated with the maximal candidate column type voting value.
4 . The computer-implemented method of claim 3 , wherein generating the overall column type prediction based at least in part on the candidate column type voting value for a candidate column type with respect to the table column further comprises:
in response to determining that the maximal candidate column type voting value fails to satisfy the candidate column type voting value threshold, generating a null overall column type prediction for the table column.
5 . The computer-implemented method of claim 1 , wherein generating a column type voting weight for a predicted column type of the plurality of predicted column types comprises:
determining the column type voting weight based at least in part on a confidence score generated by a classification machine learning model that is associated with the predicted column type.
6 . The computer-implemented method of claim 1 , wherein generating a column type voting weight for a predicted column type of the plurality of predicted column types comprises:
determining the column type voting weight based at least in part on a model weight contribution value that describes a predictive correlation between the predicted column type and a corresponding classification machine learning model.
7 . The computer-implemented method of claim 1 , wherein generating a column type voting weight for a predicted column type of the plurality of predicted column types comprises:
determining the column type voting weight based at least in part on a provider type weight contribution value that describes a predictive correlation between a provider type identifier for a reference table data object associated with the at least one reference table column and the predicted column type.
8 . The computer-implemented method of claim 1 , wherein determining a column type voting weight for a predicted column type of the plurality of predicted column types comprises:
determining the column type voting weight based at least in part on a table type weight contribution that describes a predictive correlation between a table type identifier for a reference table data object associated with the at least one reference table column and the predicted column type.
9 . The computer-implemented method of claim 1 , further comprising:
determining, by the one or more processors and via a table column clustering machine learning model and based at least in part on a plurality of table column features of the table column, a related table column cluster set for the table column, and
determining, by the one or more processors, a functional grouping of the table column based at least in part on the related table column cluster set for the table column.
10 . The computer-implemented method of claim 9 , wherein the plurality of table column features comprise at least one of a data type feature of the table column, a data pattern feature of the table column, a most frequent entity type feature of the table column, a sparsity feature of the table column, and an adjacent column name feature of the table column.
11 . The computer-implemented method of claim 9 , wherein the table column clustering machine learning model is a mean shift clustering machine learning model.
12 . The computer-implemented method of claim 9 , wherein determining the related table column cluster set for a table column comprises:
mapping the table column to a multi-dimensional clustering space characterized by the plurality of table column features, wherein the multi-dimensional clustering space comprises a plurality of table column clusters;
determining a plurality of cluster distance values for the table column with respect to the plurality of table column clusters, wherein the plurality of cluster distance values comprise a cluster distance value for a table column cluster of the plurality of table column clusters; and
determining the related table column cluster set based at least in part on a subset of the plurality of table column clusters whose cluster distance values satisfies a cluster distance value threshold.
13 . The computer-implemented method of claim 9 , wherein determining the functional grouping for a table column comprises:
identifying a group of clustered table columns having the related table column cluster set; and
determining the functional grouping based at least in part on the related table column cluster.
14 . The computer-implemented method of claim 1 , wherein the entity recognition classification machine learning model is a natural language processing machine learning model characterized by various defined entity collections.
15 . The computer-implemented method of claim 1 , wherein generating the table column pair similarity measure comprises:
generating a table column mapping for the table column in a multi-dimensional clustering space;
generating a reference table column mapping for the at least one reference table column in the multi-dimensional clustering space; and
determining the table column pair similarity measure based at least in part on the table column mapping and the reference table column mapping.
16 . The computer-implemented method of claim 1 , further comprising:
identifying, by the one or more processors, an unidentified table column set, where the overall column type prediction for a table column in the unidentified table column set describes that the table column is not associated with a candidate table column type, and
generating, by the one or more processors, an overall unidentified table column report that describes one or more unidentified table column groupings as determined based at least in part on the unidentified table column set.
17 . The computer-implemented method of claim 16 , wherein determining the one or more unidentified table column groupings comprises:
for an unidentified table column of the unidentified table column set, determining a plurality of unidentified table column features; and
determining, based at least in part on the plurality of unidentified table column features for an unidentified table column, the one or more unidentified table column groupings of the unidentified table column set.
18 . A system comprising one or more processors and at least one memory storing processor-executable instructions that, when executed by any one or more of the one or more processors, cause the one or more processors to perform operations comprising:
generating, for a table column of a selected dataset, among a plurality of datasets, associated with a null overall column type, (i) a plurality of predicted column types comprising a predicted header-based column type, a predicted data-based column type, a predicted entity-recognition-based column type, and a predicted pattern-matching-based column type, and (ii) a plurality of column type voting weights comprising a header-based column type voting weight, a data-based column type voting weight, an entity-recognition-based column type voting weight, and a pattern-matching-based column type voting weight, by:
generating, via a header-based table classification machine learning model that is (i) trained to identify columns based at least in part on column headers and (ii) characterized by a first set of decision tree data objects, and based at least in part on a table column name set for the table column, the predicted header-based column type for the table column and the header-based column type voting weight for the predicted header-based column type, wherein the header-based column type voting weight is a data value indicating a first inferred credibility score based at least in part on a subset of the first set of decision tree data objects which have selected the predicted header-based column type,
generating, via a data-based table classification machine learning model that is (i) trained to identify columns based at least in part on column values and (ii) characterized by a second set of decision tree data objects, and based at least in part on a table column value set for the table column, the predicted data-based column type for the table column and the data-based column type voting weight for the predicted data-based column type, wherein the data-based column type voting weight is a data value indicating a second inferred credibility score based at least in part on a subset of the second set of decision tree data objects which have selected the predicted data-based column type;
generating, via an entity recognition classification machine learning model and based at least in part on the table column value set, the predicted entity-recognition-based column type for the table column and the entity-recognition-based column type voting weight for the predicted entity-recognition-based column type, and
generating, via a pattern matching classification machine learning model and based at least in part on the table column value set, the predicted pattern-matching-based column type for the table column and the pattern-matching-based column type voting weight for the predicted pattern-matching-based column type,
generating, via a voting machine learning model and based at least in part on the plurality of predicted column types and the plurality of column type voting weights, an overall column type prediction for the table column;
generating, via a structural variance generation machine learning model and based at least in part on a comparison between the table column and at least one reference table column from at least one additional dataset from the plurality of datasets, a table column pair similarity measure;
generating, responsive to a determination that the table column pair similarity measure exceeds a threshold, an unidentified table column grouping comprising the table column and the at least one reference table column;
generating an emergence report indicating updated data corresponding to the table column of the selected dataset, wherein the updated data comprises (i) the unidentified table column grouping and (ii) the overall column type prediction for the table column; and
providing access to the updated data to one or more users by generating a prediction output user interface configured to display the emergence report via one or more external computing entities operable by the one or more users.
19 . The system of claim 18 , wherein generating the overall column type prediction for a table column comprises:
for a candidate column type of a plurality of candidate column types:
identifying a predicted column type set of the plurality of predicted column types for the table column that corresponds to the candidate column type,
identifying a column type voting weight set of the plurality of column type voting weights that corresponds to the predicted column type set, and
determining a candidate column type voting value for the candidate column type with respect to the table column based at least in part on the column type voting weight set; and
generating the overall column type prediction based at least in part on the candidate column type voting value for a candidate column type with respect to the table column.
20 . One or more non-transitory computer-readable storage media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
generating, for a table column of a selected dataset, among a plurality of datasets, associated with a null overall column type, (i) a plurality of predicted column types comprising a predicted header-based column type, a predicted data-based column type, a predicted entity-recognition-based column type, and a predicted pattern-matching-based column type, and (ii) a plurality of column type voting weights comprising a header-based column type voting weight, a data-based column type voting weight, an entity-recognition-based column type voting weight, and a pattern-matching-based column type voting weight, by:
generating, via a header-based table classification machine learning model that is (i) trained to identify columns based at least in part on column headers and (ii) characterized by a first set of decision tree data objects, and based at least in part on a table column name set for the table column, the predicted header-based column type for the table column and the header-based column type voting weight for the predicted header-based column type, wherein the header-based column type voting weight is a data value indicating a first inferred credibility score based at least in part on a subset of the first set of decision tree data objects which have selected the predicted header-based column type,
generating, via a data-based table classification machine learning model that is (i) trained to identify columns based at least in part on column values and (ii) characterized by a second set of decision tree data objects, and based at least in part on a table column value set for the table column, the predicted data-based column type for the table column and the data-based column type voting weight for the predicted data-based column type, wherein the data-based column type voting weight is a data value indicating a second inferred credibility score based at least in part on a subset of the second set of decision tree data objects which have selected the predicted data-based column type;
generating, via an entity recognition classification machine learning model and based at least in part on the table column value set, the predicted entity-recognition-based column type for the table column and the entity-recognition-based column type voting weight for the predicted entity-recognition-based column type, and
generating, via a pattern matching classification machine learning model and based at least in part on the table column value set, the predicted pattern-matching-based column type for the table column and the pattern-matching-based column type voting weight for the predicted pattern-matching-based column type,
generating, via a voting machine learning model and based at least in part on the plurality of predicted column types and the plurality of column type voting weights, an overall column type prediction for the table column;
generating, via a structural variance generation machine learning model and based at least in part on a comparison between the table column and at least one reference table column from at least one additional dataset from the plurality of datasets, a table column pair similarity measure;
generating, responsive to a determination that the table column pair similarity measure exceeds a threshold, an unidentified table column grouping comprising the table column and the at least one reference table column;
generating an emergence report indicating updated data corresponding to the table column of the selected dataset, wherein the updated data comprises (i) the unidentified table column grouping and (ii) the overall column type prediction for the table column; and
providing access to the updated data to one or more users by generating a prediction output user interface configured to display the emergence report via one or more external computing entities operable by the one or more users.