Machine learning-based diagram label recognition
Mechanisms are disclosed for machine learning-based diagram label recognition in connection with diagrams represented by unstructured images. An unstructured image of the diagram is received. A plurality of text blocks associated with the diagram is detected. The text blocks are associated with labels contained within the diagram. Textual contents of the text blocks and the labels are recognized. The labels are associated with corresponding edges and shapes within the diagram.
1 . A non-transitory computer-readable media storing computer-executable instructions that, when executed by a processor, perform a method for machine learning-based diagram label recognition of a graphical modeling language represented by a hand-drawn image of a diagram, the method comprising:
receiving the hand-drawn image of the diagram representing a graphical modeling language, the diagram comprising shapes, edges defined by an arrow that connects two shapes, and labels associating textual contents with a shape or an edge;
generating relation scores for a set of shape and edge candidate pairs for every text block that intersect within a text block region in the hand-drawn image, the candidate pairs comprising a text block and a related shape or edge, the generating relation scores comprising:
generating a binary mask that indicates a relative location of bounding boxes of the text block and the related shape or edge withing the text block region, the binary mask being initialized with zeros and then filled with ones for each bounding box pixel that is located within the text block region;
generating a learned visual feature representation of the text block region by assessing visual features of the text block using deep learning object detectors;
generating an encoded representation of a predicted class of a target shape or edge of a relation pair using class-specific modeling rules and conventions by determining which shape or edge types are infrequently labeled;
concatenating a combined representation of the binary mask, the learned visual feature representation, and the encoded representation of the predicted class; and
processing the combined representation by a convolutional neural network with a binary classification layer using a sigmoid function to determine the relation scores;
identifying and removing duplicate text blocks that have been matched to a same shape or edge by keeping the text block having a highest relation score;
associating the labels with corresponding edges and shapes within the hand-drawn diagram using the relation scores; and
exporting a structured representation of the diagram in a Business Process Model and Notation (BPMN)-compliant format, wherein the structured representation preserves the association between the labels and diagram elements.
2 . A method for machine learning-based diagram label recognition of a graphical modeling language represented by a hand-drawn image of a diagram, the method comprising:
receiving the hand-drawn image of the diagram representing a graphical modeling language, the diagram comprising shapes, edges defined by an arrow that connects two shapes, and labels associating textual contents with a shape or an edge;
generating relation scores for a set of shape and edge candidate pairs for every text block that intersect within a text block region in the hand-drawn image, the candidate pairs comprising a text block and a related shape or edge, the generating relation scores comprising:
generating a binary mask that indicates a relative location of bounding boxes of the text block and the related shape or edge withing the text block region, the binary mask being initialized with zeros and then filled with ones for each bounding box pixel that is located within the text block region;
generating a learned visual feature representation of the text block region by assessing visual features of the text block using deep learning object detectors;
generating an encoded representation of a predicted class of a target shape or edge of a relation pair using class-specific modeling rules and conventions by determining which shape or edge types are infrequently labeled;
concatenating a combined representation of the binary mask, the learned visual feature representation, and the encoded representation of the predicted class; and
processing the combined representation by a convolutional neural network with a binary classification layer using a sigmoid function to determine the relation scores;
identifying and removing duplicate text blocks that have been matched to a same shape or edge by keeping the text block having a highest relation score;
associating the labels with corresponding edges and shapes within the hand-drawn diagram using the relation scores; and
exporting a structured representation of the diagram in a Business Process Model and Notation (BPMN)-compliant format, wherein the structured representation preserves the association between the labels and diagram elements.
3 . A system for machine learning-based diagram label recognition of a graphical modeling language represented by a hand-drawn image of a diagram, the system comprising:
at least one processor;
and at least one non-transitory memory storing computer executable instructions that when executed by the at least one processor cause the system to carry out actions comprising:
receiving the hand-drawn image of the diagram representing a graphical modeling language, the diagram comprising shapes, edges defined by an arrow that connects two shapes, and labels associating textual contents with a shape or an edge;
associating the labels with corresponding edges and shapes within the hand-drawn diagram using the relation scores; and
exporting a structured representation of the diagram in a Business Process Model and Notation (BPMN)-compliant format, wherein the structured representation preserves the association between the labels and diagram elements.