Techniques for extracting links and connectivity from schematic diagrams
In example embodiments, techniques are provided for using a combination of multiple ML models and signal processing to extract links and connectivity from a schematic diagram in an image-only format. A first ML model (i.e. link segmenter) may produce a first set of predictions about the positions of link segments in the schematic diagram (e.g., in the form of a segmentation map). A second ML model (i.e. keypoint detector) may produce a second set of predictions about starting and stopping points of link segments in the schematic diagram (e.g., in the form of one or more heatmaps). A signal processing module may combine the first set of predictions and the second set of predictions to produce a description of links and connectivity they provide (e.g., combining the segmentation map with data from the one or more heatmaps). The results of the combining may be saved as a graph connectivity matrix.
1. A method for extracting links and connectivity from a schematic diagram in an image-only format, comprising:
producing, using a first machine learning (ML) model executing on one or more computing devices, a first set of predictions about positions of link segments in the schematic diagram;
producing, using a second ML model executing on the one or more computing devices, a second set of predictions about starting and stopping points of link segments in the schematic diagram;
performing an optimization, by a signal processing module executing on the one or more computing devices, to combine the first set of predictions about the positions of link segments and the second set of predictions about starting and stopping points of link segments to produce a description of links and connectivity they provide between symbols in the schematic diagram, wherein the optimization associates portions of link segments with starting and stopping points, applies a cost to each possible association, and selects a set of associations that provides a globally optimal cost minimum; and
outputting the description of links and connectivity in a machine-readable format.
2. The method of claim 1 , wherein the first set of predictions about the positions of link segments comprises a segmentation map that, for each position in the schematic diagram, indicates a probability the position in the schematic diagram is part of a link.
3. The method of claim 1 , wherein the first ML model that produces the first set of predictions about the positions of link segments is structured according to a convolutional neural network architecture adapted for semantic segmentation.
4. The method of claim 1 , wherein the first ML model consumes, in addition to the schematic diagram in the image-only format, a mask of detected symbols in the schematic diagram.
5. The method of claim 1 , wherein the second set of predictions about the positions of link segments is based on one or more heatmaps that, for each position in the schematic diagram, indicates a probability the position in the schematic diagram is a connection point.
6. The method of claim 1 , wherein producing the second set of predictions about starting and stopping points of link segments, further comprises:
classifying the starting and stopping points as either an associated connection point or a floating connection point,
wherein the second set of predictions about starting and stopping points of link segments is based on a first heatmap that, for each position in the schematic diagram, indicates a probability the position in the schematic diagram is an associated connection point, and a second heatmap that, for each position in the schematic diagram, indicates a probability the position in the schematic diagram is a floating connection point.
7. The method of claim 1 , wherein the second ML model that produces the second set of predictions about starting and stopping points of link segments is structured according to a deep neural network architecture adapted for anchorless object detection.
8. The method of claim 1 , wherein the second ML model consumes, in addition to the schematic diagram in the image-only format, a mask of detected symbols in the schematic diagram and one or more masks of detected text regions in the schematic diagram.
9. The method of claim 1 , wherein the description of links and connectivity comprises a graph connectivity matrix that indicates, for each symbol in the schematic diagram, which other symbols it is connected to by a link.
10. The method of claim 1 , wherein the performing an optimization comprises:
vectorizing the link segments in the first set of predictions and inserting nodes,
wherein and the portions of link segments that the optimization associates with starting and stopping points are the nodes inserted into the first set of predictions.
11. The method of claim 1 , further comprising:
predicting, using the first ML model, a link type based on a visual appearance of one or more line segments depicting the link in the schematic diagram; and
associating at least one link in the description of links and connectivity with a respective link type.
12. The method of claim 1 , further comprising:
predicting, using the first ML model, a flow direction based on a visual appearance of an arrow of a line segment depicting the link in the schematic diagram; and
associating at least one link in the description of links and connectivity with a respective flow direction.
13. The method of claim 1 , wherein the machine-readable format is a JavaScript Object Notation (JSON) file.
14. A computing device comprising:
a display screen;
a processor; and
a memory coupled to the processor and configured to store a schematic diagram in an image-only format and a schematic diagram data extraction application that includes a first machine learning (ML) model and a second ML model, the schematic diagram data extraction application when executed operable to:
predict, using the first ML model, a set of link segments to produce a segmentation map that indicates positions of link segments;
predict, using the second ML model, a set of discrete connection points from the schematic diagram to produce one or more heatmaps that indicate positions of starting and stopping points of link segments;
perform an optimization to combine the segmentation map and data from the one or more heatmaps to produce a description of links and connectivity they provide between symbols in the schematic diagram, wherein the optimization associates portions of link segments from the segmentation map with the starting and stopping points of link segments from the one or more heatmaps, applies a cost to each possible association, and selects a set of associations that provides a globally optimal cost minimum; and
output the description of links and connectivity in a machine-readable format.
15. The computing device of claim 14 , wherein the segmentation map, for each position in the schematic diagram, indicates a probability the position in the schematic diagram is part of a link, and the one or more heatmaps, for each position in the schematic diagram, indicates a probability the position in the schematic diagram is a connection point.
16. The computing device of claim 14 , wherein the description of links and connectivity comprises a graph connectivity matrix which indicates, for each symbol in the schematic diagram, which other symbols it is connected to by a link.
17. A non-transitory computing device readable medium having instructions stored thereon, the instructions when executed by one or more computing devices operable to:
produce a first set of predictions about the positions of link segments in the schematic diagram;
produce a second set of predictions about starting and stopping points of link segments in the schematic diagram;
combine the first set of predictions and second set of predictions to produce a description of links and connectivity they provide between symbols in the schematic diagram by
vectorizing the link segments in the first set of predictions and inserting nodes, and
performing an optimization to associate the nodes inserted into the first set of predictions with starting and stopping points from the second set of predictions, the optimization to associate a cost to each possible association and the optimization to select a set of associations that provides a globally optimal cost minimum; and
output the description of links and connectivity in a machine-readable format.
18. The non-transitory electronic-device readable medium of claim 17 , wherein the first set of predictions about the positions of link segment in the schematic diagram are produced using a first machine learning (ML) model and the second set of predictions about starting and stopping points of link segments in the schematic diagram are produced using a second ML model.
19. The non-transitory electronic-device readable medium of claim 17 , wherein the first set of predictions about the positions of link segments comprises a segmentation map that, for each position in the schematic diagram, indicates a probability the position in the schematic diagram is part of a link, and the second set of predictions about the positions of link segments is based on one or more heatmaps that, for each position in the schematic diagram, indicates a probability the position in the schematic diagram is a connection point.
20. The non-transitory electronic-device readable medium of claim 19 , wherein the description of links and connectivity comprises a graph connectivity matrix which indicates, for each symbol in the schematic diagram, which other symbols it is connected to by a link.
21. A method for extracting links and connectivity from a schematic diagram in an image-only format, comprising:
producing, using a first machine learning (ML) model executing on one or more computing devices, a first set of predictions about positions of link segments in the schematic diagram;
producing, using a second ML model executing on the one or more computing devices, a second set of predictions about starting and stopping points of link segments in the schematic diagram;
combining, by software executing on the one or more computing devices, the first set of predictions and second set of predictions to produce a description of links and connectivity they provide between symbols in the schematic diagram by
vectorizing the link segments in the first set of predictions and inserting nodes, and
performing an optimization to associate the nodes inserted into the first set of predictions with starting and stopping points from the second set of predictions, the optimization to associate a cost to each possible association and the optimization to select a set of associations that provides a globally optimal cost minimum; and
outputting the description of links and connectivity in a machine-readable format.
22. The method of claim 21 , wherein the first set of predictions about the positions of link segments comprises a segmentation map that, for each position in the schematic diagram, indicates a probability the position in the schematic diagram is part of a link, and the second set of predictions about the positions of link segments is based on one or more heatmaps that, for each position in the schematic diagram, indicates a probability the position in the schematic diagram is a connection point.
23. The method of claim 21 , wherein the description of links and connectivity comprises a graph connectivity matrix that indicates, for each symbol in the schematic diagram, which other symbols it is connected to by a link.