Classifying elements and predicting properties in an infrastructure model through prototype networks and weakly supervised learning
In example embodiments, a software service may employ a neural network to learn a non-linear mapping that transforms element features into embeddings. The neural network may be trained to distribute the embeddings in multi-dimensional embedding space, such that distance between the embeddings is meaningful to the class or category classification, or property prediction, task at hand. The neural network may be trained using weakly supervised machine learning, using weakly labeled infrastructure models. Embeddings for groups may be used to determine prototypes. Elements of an infrastructure model may be classified into classes or categories, or their properties predicted, as the case may be, by finding a nearest prototype.
1 . A method for reclassifying an element of an infrastructure model that is misclassified or is missing a class, category or property to enable analytics, comprising:
accessing one or more training infrastructure models that include a plurality of elements that model individual portions of infrastructure, wherein infrastructure is a physical structure or object to be built;
during training, for the one or more training infrastructure models,
selecting, by software executing on one or more computing devices, a plurality of groups of elements associated with different classes, categories or properties from the training infrastructure models, wherein the selection is with equal probability or weighted;
generating, by a neural network of the software, a non-linear feature-embedding mapping that maps features of elements of the groups of the training infrastructure models to embeddings, wherein the features are derived from metadata describing geometric, contextual or temporal aspects of the portion of infrastructure modeled by the elements, and the embeddings are multi-dimensional numerical representations distributed in multi-dimensional embedding space such that distance between the embeddings is meaningful to a class, category or property of the elements of the infrastructure models,
determining, by the software, prototypes for each of the plurality of groups of elements based on the embeddings for the elements of a respective group, wherein a prototype is a single representation in the multi-dimensional embedding space that embeddings of a group cluster around,
identifying embeddings learned by the neural network that are outliers from the determined prototypes and correcting the outliers by updating a class, category or property of the elements in the training infrastructure models corresponding to the outliers, and
refining, by the software, the non-linear feature-embedding mapping based on the corrected outliers by repeating the selecting, generating, determining prototypes, and identifying embeddings that are outliers and correcting the outliers, until a condition is met; and
during inference, for the infrastructure model that includes misclassified elements or elements that are missing classes, categories or properties,
determining, by the software, features for an element of the infrastructure model,
applying, by the software, the non-linear feature-embedding mapping to the determined features to produce an embedding for the element,
predicting, by the software, a class, category or property of the element based on distance in multi-dimensional embedding space between the embedding for the element and one or more prototypes of the infrastructure model,
comparing, by the software, the predicted class, category or property of the element based on the distance between the embedding for the element and the one or more prototypes to a currently associated, category or property of the element,
providing, by the software, an output that indicates that the element is misclassified or that the element is missing the class, category or property,
reclassifying, by the software, the element to the predicted class, category or property, and
executing one or more software-based analytical tools on the infrastructure model with the reclassified element to provide a dashboard for monitoring project performance or measuring impact of design changes to the infrastructure.
2 . The method of claim 1 , wherein the output includes an indication that the element is misclassified, and a suggestion of reclassifications to the predicted class, category or property.
3 . The method of claim 1 , wherein the output includes an indication the element is missing a class, category or property, and a suggestion of classification to the predicted class, category or property.
4 . The method of claim 1 , wherein the output includes an embedding-based visualization in which the embedding of the element and embeddings of other elements of the plurality of groups are plotted as points in multi-dimensional embedding space.
5 . The method of claim 4 , wherein the providing further comprises:
reducing dimensionality of the embeddings to produce a set of points that represent elements;
selecting a mapping of visual features of points to aspects of the elements; and
plotting, by the software, the set of points with the selected visual features to produce the embedding-based visualization.
6 . The method of claim 4 , wherein the visual features include colors, and the embedding-based visualization maps class, category or property to colors, such that points having a common class, category or property share a common color.
7 . The method of claim 1 , wherein the determined features are represented in k-dimensional space and the multi-dimensional embedding space is n-dimensional space, with n different from k.
8 . The method of claim 7 , wherein the output includes an embedding-based visualization in which the embedding of the element and embeddings of other elements of the plurality of groups are plotted as points in x-dimensional space, and x<n.
9 . The method of claim 1 , further comprising:
sampling, by the software, training elements from the plurality of groups to produce a training set of features for the training elements;
providing, by the software, the training set of features for the training elements as input to the neural network; and
for each of a plurality of elements of the groups,
calculating a probability of an embedding for the element belonging to each group based on a distance in multi-dimensional embedding space between the embedding and the prototype for each of the groups, and determining a predicted group from the probability of the element belonging to each group, and
determining a loss based on a comparison of the predicted group and a true group, the loss used to provide feedback to the neural network.
10 . A method for reclassifying an element of an infrastructure model that is misclassified or is missing a class, category or property, comprising:
training, by software executing on one or more computing devices, a neural network to learn a non-linear feature-embedding mapping that maps features of elements of one or more training infrastructure models to embeddings, wherein the features are derived from metadata describing geometric, contextual or temporal aspects of the portion of infrastructure modeled by the elements, infrastructure is a physical structure or object to be built, and the embeddings are multi- dimensional numerical representations distributed in multi-dimensional embedding space such that distance between the embeddings is meaningful to a class, category or property of the elements of the infrastructure models, wherein the training includes
selecting, by the software, a plurality of groups of elements associated with different classes, categories or properties from the training infrastructure models, wherein the selection is with equal probability or weighted,
sampling, by the software, elements from the plurality of groups to produce a set of features for the elements,
providing, by the software, the set of features for the elements as input to the neural network to generate the non-linear mapping of features to embeddings,
for each of the plurality of groups,
determining, by the software, using a first subset of the embeddings, a prototype for the group, the prototype being a single representation in the multi-dimensional embedding space that the embeddings of the group cluster around, and
determining, by the software, using a second subset of the embeddings, a predicted group for a plurality of elements based on a distance in multi-dimensional embedding space between the embedding for a respective element and the prototypes for each of the plurality of groups,
determining, by the software, a loss based on a comparison of the predicted group and a true group for the plurality of elements, the loss used to provide feedback to the neural network,
identifying embeddings learned by the neural network that are outliers from the determined prototypes and correcting the outliers by updating a class, category or property of the elements in the training infrastructure models corresponding to the outliers;
repeating the sampling, providing, determining the prototype, determining the predicted group, determining the loss, and the identifying and correcting, over the course of a plurality of training steps to train the neural network; and
during inference, for the infrastructure model that includes the element that is misclassified is missing the class, category or property,
using the trained neural network to reclassify the element, and
executing one or more software-based analytical tools on the infrastructure model with the reclassified element to provide a dashboard for monitoring project performance or measuring impact of design changes to the infrastructure.
11 . The method of claim 10 , further comprising:
using the trained infrastructure model to produce an embedding-based visualization in which embeddings of elements are plotted as points in multi-dimensional embedding space.
12 . A non-transitory computer-readable medium having instructions stored thereon, the instructions when executed on one or more processors of one or more computing devices being operable to:
access one or more training infrastructure models that include a plurality of elements that model individual portions of infrastructure, wherein infrastructure is a physical structure or object to be built;
during training, for one or more training infrastructure models,
select a plurality of groups of elements associated with different classes, categories or properties from the training infrastructure models, wherein the selection is with equal probability or weighted;
generate, by a neural network, a non-linear feature-embedding mapping that maps features of elements of the training infrastructure models to embeddings, wherein the features are derived from metadata describing geometric, contextual or temporal aspects of the portion of infrastructure modeled by the elements, and the embeddings are multi-dimensional numerical representations distributed in multi-dimensional embedding space such that distance between the embeddings is meaningful to a class, category or property of the elements of the infrastructure models,
determine prototypes for a each of the plurality of groups of elements based on the embeddings for the elements of a respective group learned by the neural network, wherein a prototype is a single representation in the multi-dimensional embedding space that embeddings of a group cluster around,
identify embeddings learned by the neural network that are outliers from the determined prototypes and correct the outliers by updating a class, category or property of the elements in the training infrastructure models corresponding to the outliers,
refine the non-linear feature-embedding mapping learned by the neural network based on the corrected outliers by repeating the operations to select, generate, determine prototypes, and identify embeddings that are outliers and correct the outliers, until a condition is met; and
provide the non-linear feature-embedding mapping for use in reclassify elements of an infrastructure model that are misclassified or are missing a class, category or property.
13 . The non-transitory computer-readable medium of claim 12 , wherein the instructions are further operable to:
sample training elements from the plurality of groups to produce a training set of features for the training elements;
provide the training set of features for the training elements as input to the neural network; and
for each of a plurality of elements of the groups,
calculate a probability of an embedding for the element belonging to each group based on a distance in multi-dimensional embedding space between the embedding and the prototype for each of the groups, and determine a predicted group from the probability of the element belonging to each group, and
determine a loss based on a comparison of the predicted group and a true group, the loss used to provide feedback to the neural network.
14 . The non-transitory computer-readable medium of claim 12 , wherein the instructions to provide include instructions operable to:
display an indication that the elements are misclassified or are missing a class, category or property, and a suggestion of classification to a predicted class, category or property.
15 . The non-transitory computer-readable medium of claim 12 , wherein the instructions to provide include instructions operable to:
display an embedding-based visualization for at least one of the elements that are misclassified or are missing a class, wherein the embedding-based visualization depicts the embedding of the element and embeddings of other elements plotted as points in multi-dimensional embedding space.
16 . A computing device comprising:
one or more processors; and
one or more memories storing executable instructions for training a neural network to reclassify an element of an infrastructure model that is misclassified or is missing a class, category or property, the instructions when executed operable to train the neural network by:
for one or more training infrastructure models infrastructure models that include a plurality of elements that model individual portions of infrastructure, wherein infrastructure is a physical structure or object to be built,
selecting a plurality of groups of elements associated with different classes, categories or properties from the training infrastructure models, wherein the selection is with equal probability or weighted,
generating, using the neural network, a non-linear feature-embedding mapping that maps features of elements of the training infrastructure models to embeddings, wherein the features are derived from metadata describing geometric, contextual or temporal aspects of the portion of infrastructure modeled by the elements, and the embeddings are multi-dimensional numerical representations distributed in multi-dimensional embedding space such that distance between the embeddings is meaningful to a class, category or property of the elements of the infrastructure models,
determining prototypes for each of the plurality of groups of elements based on the embeddings for the elements of a respective group, wherein a prototype is a single representation in the multi-dimensional embedding space that embeddings of a group cluster around,
identifying embeddings that are outliers from the determined prototypes and correcting the outliers by updating a class, category or property of the elements in the training infrastructure models corresponding to the outliers, and
refining the non-linear feature-embedding mapping based on the corrected outliers; and
providing the non-linear feature-embedding mapping for use in reclassify elements of an infrastructure model that are misclassified or are missing a class, category or property.
17 . The computing device of claim 16 , wherein the instructions are further operable to train the neural network by:
sampling training elements from the plurality of groups to produce a training set of features for the training elements;
providing the training set of features for the training elements as input to the neural network; and
for each of a plurality of elements of the groups,
calculating a probability of an embedding for the element belonging to each group based on a distance in multi-dimensional embedding space between the embedding and the prototype for each of the groups, and determining a predicted group from the probability of the element belonging to each group, and
determining a loss based on a comparison of the predicted group and a true group, the loss used to provide feedback to the neural network.