Techniques for extracting associations between text labels and symbols and links in schematic diagrams
In example embodiments, techniques are provided that use two different ML models (a symbol association ML model and a link association ML model), one to extract associations between text labels and one to extract associations between symbols and links, in a schematic diagram (e.g., P&ID) in an image-only format. The two models may use different ML architectures. For example, the symbol association ML model may use a deep learning neural network architecture that receives for each possible text label and symbol pair both a context and a request, and produces a score indicating confidence the pair is associated. The link association ML model may use a gradient boosting tree architecture that receives for each possible text label and link pair a set of multiple features describing at least the geometric relationship between the possible text label and link pair and produces a score indicating confidence the pair is associated.
1. A method for extracting associations between text labels and symbols or links in a schematic diagram in an image-only format, comprising:
accessing, by an application executing on one or more computing devices, the schematic diagram and a set of texts labels, a set of symbols, and a set of links that have been extracted from the schematic diagram;
determining possible text label and symbol pairs from the set of texts labels and the set of symbols;
for each possible text label and symbol pair, applying to a symbol association machine learning (ML) model a context that describes a region of the schematic diagram surrounding the possible text label and symbol pair and a request that precisely designates the possible text label and symbol pair, to produce a text label-symbol score indicating confidence in association;
selecting, by the application, associations between text labels and symbols based on the text label-symbol scores;
determining possible text label and link pairs from the set of texts labels and the set of links;
for each possible text label and link pair, applying to a link association ML model of the application a set of multiple features describing at least the geometric relationship between the possible text label and link pair, to produce a text label-link score indicating confidence in association;
selecting, by the application, associations between text labels and symbols and text labels and links based on the text label-link scores; and
outputting the selected associations between text labels and symbols and text labels and links in a machine-readable format.
2. The method of claim 1 , wherein the symbol association ML model and the link association ML model use different ML architectures.
3. The method of claim 2 , wherein the symbol association ML model uses a deep learning neural network architecture, and the link association ML model uses a gradient boosting tree architecture.
4. The method of claim 1 , wherein, the applying to the symbol association ML model further comprises:
extracting as the context an image of the region, wherein the region encloses at least a bounding box of the text label and a bounding box of the symbol;
providing the context to an image encoder to produce an encoded context;
determining coordinates of the bounding box of the text label and the bounding box of the symbol; and
providing the coordinates to a coordinate encoder to produce an encoded request.
5. The method of claim 4 , wherein the applying to the symbol association ML model further comprises:
providing the encoded context and the encoded request to a merge and classification neural network that includes a set of fully connected layers that is trained to produce the text label-symbol score.
6. The method of claim 1 , wherein the determining possible text label and symbol pairs further comprises:
automatically associating a given text label and a given symbol when a bounding box of the given text label is within a bounding box of the symbol; and
excluding the given text label and given symbol pair from the possible text label and symbol pairs.
7. The method of claim 1 , wherein the determining possible text label and symbol pairs further comprises:
automatically associating a given text label and a given symbol when the bounding box of the given text label is under a threshold distance of the bounding box of the symbol, and the given text label and the given symbol mutually see each other as the closest region; and
excluding the given text label and given symbol pair from the possible text label and symbol pairs.
8. The method of claim 1 , wherein the determining possible text label and symbol pairs further comprises:
determining a distance from a bounding box of a given text label to a bounding box of a given symbol is greater than a maximum distance; and
excluding the given text label and given symbol pair from the possible text label and symbol pairs.
9. The method of claim 1 , wherein the symbol association ML model uses a loss function during training that includes a penalty to associations that is quadratically increasing with distance.
10. The method of claim 1 , wherein the applying to the link association ML model further comprises:
computing a multidimensional feature vector that represents the multiple features; and
applying the possible text label and link pair and the multidimensional feature vector to the link association ML model to produce an initial text label-link score.
11. The method of claim 10 , wherein the computing the multidimensional feature vector further comprises:
encoding two or more features into a single value included in the multidimensional feature vector to indicate dependency between features.
12. The method of claim 10 , wherein the determining possible text label and link pairs further comprises:
determining a distance from a given link to a bounding box of a given text label is greater than a maximum distance; and
excluding the given text label and given link pair from the possible text label and link pairs.
13. The method of claim 10 , wherein the applying to the link association ML model further comprises:
determining a given text label is already associated with greater than a limit for number of links; and
preventing association of the given text label and given link.
14. The method of claim 1 , wherein the multiple features comprise one or more of a link type feature, a link to text label distance feature, a relative position between the text label and the link feature, an orientation of the text label feature, an orientation of the link feature, a projection of a baseline of the text label onto the link feature, a projection angle between the link and a baseline of the text label feature, a collision cost feature or an aspect ratio of a bounding box that includes both the text label and the link feature.
15. The method of claim 1 , wherein the schematic diagram is a piping and instrumentation diagram (P&ID).
16. A computing device comprising:
a display screen;
a processor; and
a memory coupled to the processor and configured to store a schematic diagram in an image-only format and a schematic diagram data extraction application that includes a symbol association machine learning (ML) model having a first ML architecture and a link association ML model having a second, different ML architecture, the schematic diagram data extraction application when executed operable to:
access a schematic diagram and a set of texts labels, a set of symbols, and a set of links that have been extracted from the schematic diagram,
determine possible text label and symbol pairs from the set of texts labels and the set of symbols,
for each possible text label and symbol pair, apply a description of the possible text label and symbol pair to the symbol association ML model to produce a text label to symbol association,
determine possible text label and link pairs from the set of texts labels and the set of links,
for each possible text label and link pair, apply a description of the possible text label and link pair to the link association ML model to produce a text label to link association, and
output the text label to symbol associations and the text label to link associations in a machine-readable format.
17. A non-transitory computing device readable medium having instructions stored thereon, the instructions when executed by one or more computing devices operable to:
access a schematic diagram and a set of texts labels, set of symbols, and set of links that have been extracted from the schematic diagram;
determine possible text label and symbol pairs from the set of texts labels and the set of symbols;
for each possible text label and symbol pair, apply to a symbol association machine learning (ML) model a context that describes a region of the schematic diagram surrounding the possible text label and symbol pair and a request that precisely designates the possible text label and symbol pair to produce a text label to symbol association;
for each possible text label and link pair, apply to a link association ML model a set of multiple features describing at least the geometric relationship between the possible text label and link pair to produce a text label to link association; and
output the text label to symbol associations and the text label to link associations in a machine-readable format.
18. The non-transitory electronic-device readable medium of claim 17 , wherein the symbol association ML model uses a deep learning neural network architecture, and the link association ML model uses a gradient boosting tree architecture.
19. The non-transitory electronic-device readable medium of claim 17 , wherein the instructions that when executed are operable to apply the symbol association ML model further comprise instructions that when executed are operable to:
extract as the context as an image to the region, wherein the region encloses at least a bounding box of the text label and a bounding box of the symbol;
provide the context to an image encoder to produce an encoded context;
determine coordinates of the bounding box of the text label and the bounding box of the symbol;
provide the coordinates to a coordinate encoder to produce an encoded request; and
provide the encoded context and the encoded request to a merge and classification neural network that includes a set of fully connected layers that is trained to produce the text label to symbol association.
20. The non-transitory electronic-device readable medium of claim 17 , wherein the instructions that when executed are operable to apply the link association ML model further comprise instructions that when executed are operable to:
compute a multidimensional feature vector that represents the multiple features;
apply the possible text label and link pair and the multidimensional feature vector to the link association ML model to produce an initial text label to link association; and
adjust the initial text label to link association when one or more criteria are met to produce the text label to link association.