IP Library › Granted Patent US 12,347,038
Granted Patent B2
US 12,347,038 · App. 17/027,829 · Granted Jul 1, 2025

Crack detection, assessment and visualization using deep learning with 3D mesh model

Inventors: Zheng Yi Wu (Watertown, CT); Rony Kalfarisi (North Sumatera, ID); Ken Soh (Singapore, SG)
Assignee: Bentley Systems, Incorporated
G06T19/006G01C11/02G01C11/04G06F18/2148G06V10/22G06V10/44G06V20/20G06T2219/004
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,347,038
App. No.
17/027,829
Granted
Jul 1, 2025
Kind
B2
Abstract

In various example embodiments, techniques are provided for crack detection, assessment and visualization that utilize deep learning in combination with a 3D mesh model. Deep learning is applied to a set of 2D images of infrastructure to identify and segment surface cracks. For example, a Faster region-based convolutional neural network (Faster-RCNN) may identify surface cracks and a structured random forest edge detection (SFRED) technique may segment the identified surface cracks. Alternatively, a Mask region-based convolutional neural network (Mask-RCNN) may identify and segment surface cracks in parallel. Photogrammetry is used to generate a textured three-dimensional (3D) mesh model of the infrastructure from the 2D images. A texture cover of the 3D mesh model is analyzed to determine quantitative measures of identified surface cracks. The 3D mesh model is displayed to provide a visualization of identified surface cracks and facilitate inspection of the infrastructure.

Claims (32)

1. A method comprising:

acquiring a set of two-dimensional (2D) images of infrastructure;

identifying surface cracks in the 2D images and segmenting the identified surface cracks in the 2D images, by a trained Mask Region-Based Convolutional Neural Network (Mask-RCNN) executing on one or more electronic devices, to produce segmented 2D images having pixels that are divided into crack pixels that are part of identified surface cracks, and non-crack pixels that are not part of the identified surface cracks;

extracting, by a photogrammetry process executing on the one or more electronic devices, geometry information from the segmented 2D images produced by the Mask-RCNN and generating a three-dimensional (3D) mesh model of the infrastructure from the geometry information;

extracting, by the photogrammetry process, textures from the segmented 2D images produced by the Mask-RCNN and draping the textures on the 3D mesh model as a texture cover to form a textured 3D mesh model of the infrastructure;

indicating the identified surface cracks in the textured 3D mesh model by showing the crack pixels in the texture cover of the textured 3D mesh model with a contrasting color;

displaying, in a user interface on a display screen of the one or more electronic devices, the textured 3D mesh model to provide a visualization of identified surface cracks;

determining, by an analysis process executing on the one or more electronic devices, one or more quantitative measures of identified surface cracks in pixels based on the texture cover of the textured 3D mesh model;

determining a conversion factor between a number of pixels in the texture cover of the textured 3D mesh model and a distance in the infrastructure;

converting the one or more quantitative measures of identified surface cracks in pixels to one or more quantitative measures for the infrastructure by applying the conversion factor; and

displaying, in a user interface on a display screen, an indication of the one or more quantitative measures of identified surface cracks for the infrastructure.

2. The method of claim 1 , wherein the determining one or more quantitative measures further comprises:

fusing the texture cover to produce a fused texture cover that depicts a surface of the infrastructure without any overlap.

3. The method of claim 1 , wherein the one or more quantitative measures include at least one of crack area, length or width.

4. The method of claim 1 , wherein the set of 2D images is acquired using a handheld camera and/or a camera of an unmanned aerial vehicle (UAV), and the photogrammetry process is a structure from motion (SFM) photogrammetry process.

5. The method of claim 1 , further comprising:

training a Mask-RCNN using transfer learning which pretrains a network with a base dataset and task and then transfers learned features to a second network that is fine-tuned with a crack image dataset to produce the trained Mask-RCNN.

6. The method of claim 5 , wherein the crack image dataset includes a plurality of 2D images of infrastructure captured during inspections and labeled with identified cracks.

7. A non-transitory electronic device-readable medium having executable instructions stored thereon, the instructions when executed by one or more processors being operable to:

acquire a set of two-dimensional (2D) images of infrastructure from a handheld camera and/or a camera of an unmanned aerial vehicle (UAV);

identify surface cracks in the 2D images and segment the identified surface cracks in the 2D images, using a trained Mask Region-Based Convolutional Neural Network (Mask-RCNN), to produce segmented 2D images having pixels that are divided into crack pixels that are part of identified surface cracks, and non-crack pixels that are not part of the identified surface cracks;

extract geometry information from the segmented 2D images produced by the Mask-RCNN and generate a three-dimensional (3D) mesh model of the infrastructure from the geometry information;

extract textures from the segmented 2D images produced by the Mask-RCNN and drape the textures on the 3D mesh model as a texture cover to form a textured 3D mesh model of the infrastructure;

indicate the identified surface cracks in the textured 3D mesh model by showing the crack pixels in the texture cover of the textured 3D mesh model with a contrasting color;

store the textured 3D mesh model;

determine one or more quantitative measures of identified surface cracks in pixels based on the texture cover of the textured 3D mesh model;

determine a conversion factor between a number of pixels in the texture cover of the textured 3D mesh model and a distance in the infrastructure;

convert the one or more quantitative measures of identified surface cracks in pixels to one or more quantitative measures for the infrastructure by applying the conversion factor; and

store an indication of the one or more quantitative measures of identified surface cracks for the infrastructure.

8. The non-transitory electronic device-readable medium of claim 7 , wherein the instructions to determine one or more quantitative measures include instructions that when executed are operable to fuse the texture cover to produce a fused texture cover that depicts a surface of the infrastructure without any overlap.

9. The non-transitory electronic device-readable medium of claim 7 , wherein the instructions when executed are further operable to:

train a deep learning process using transfer learning which pretrains a network with a base dataset and task and then transfers learned features to a second network that is fine-tuned with a crack image dataset to produce a final model, wherein the crack image dataset includes a plurality of 2D images of infrastructure captured during inspections that are labeled with identified cracks.

Assignments (2)
SECURITY INTEREST Recorded Oct 25, 2024
From: BENTLEY SYSTEMS, INCORPORATED
To: PNC BANK, NATIONAL ASSOCIATION
Reel/Frame 069268/0042 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 22, 2020
From: WU, ZHENG YI; KALFARISI, RONY; SOH, KEN
To: BENTLEY SYSTEMS, INCORPORATED
Reel/Frame 053843/0211 →
Continuity (1)
Related Publication 20220092856A1 · Mar 24, 2022
References Cited (66)
US 9196048B2 · Jahanshahi et al. · 2015 [cited by applicant]
US 9964468B1 · Wu et al. · 2018 [cited by applicant]
US 9983776B1 · Wu et al. · 2018 [cited by applicant]
US 10295435B1 · Wu et al. · 2019 [cited by applicant]
US 10650588B2 · Hazeghi et al. · 2020 [cited by applicant]
US 20130034305A1 · Jahanshahi et al. · 2013 [cited by applicant]
US 20180336683A1 · Feng · 2018 [cited by examiner]
US 20200166907A1 · Frederick · 2020 [cited by examiner]
US 20200284686A1 · Li · 2020 [cited by examiner]
US 20200342209A1 · Li · 2020 [cited by examiner]
US 20210082102A1 · Denigan, III · 2021 [cited by examiner]
Abdel-Qader, Ikhals, et al., “Analysis of Edge-Detection Techniques for Crack Identification in Bridges,” American Society of Civil Engineers, ASCE, Journal of Computing in Civil Engineering, vol. 17, Issue 4, Oct. 1, 2… [cited by applicant]
Abdel-Qader, Ikhals, et al., “PCA-Based Algorithm for Unsupervised Bridge Crack Detection,” Elsevier Ltd, Elsevier, Science Direct, Advances in Engineering Software, vol. 37, Issue 12, Dec. 2006, pp. 771-778. [cited by applicant]
Abe, Satoshi, et al., “A High-Speed Image Processor for Detection of Pavement Cracks,” In Proceedings of the International Association for Pattern Recognition, IAPR, Workshop on Machine Vision Applications, MVA'92, Toky… [cited by applicant]
Adhikari, R. S., et al., “Image-Based Retrieval of Concrete Crack Properties,” The International Association for Automation and Robotics in Construction, I.A.A.R.C., 2012 Proceedings of the 29th ISARC International Symp… [cited by applicant]
Alaknanda, A., et al., “Flaw Detection in Radiographic Weld Images Using Morphological Approach,” Elsevier Ltd, Elsevier, Science Direct, NDT& E International, vol. 39, Jul. 26, 2005, pp. 29-33. [cited by applicant]
Amano, Isao, et al., “Faster, Safer, Smarter: Automatic Crack Detection Methods on Larger Scale Concrete Infrastructure,” The American Society for Nondestructive Testing, Inc., From 2016 Digital Imaging-Improving Inspec… [cited by applicant]
Amhaz, Rabih, et al., “Automatic Crack Detection on 2D Pavement Images: An Algorithm Based on Minimal Path Selection,” IEEE, IEEE Transactions on Intelligent Transportation Systems, vol. 17, Issue 10, Sep. 20, 2016, pp.… [cited by applicant]
Behzadan, Amir H., et al., “Augmented Reality Visualization: A Review of Civil Infrastructure System Applications,” Elsevier Ltd, Elsevier, Science Direct, Advanced Engineering Informatics, vol. 29, Apr. 7, 2015, pp. 25… [cited by applicant]
Bu, G.P., et al., “Crack Detection Using a Texture Analysis-based Technique for Visual Bridge Inspection,” eJSE International, Special Issue: Electronic Journal of Structural Engineering, vol. 14, Issue 1, Jan. 2015, pp… [cited by applicant]
Cha, Young-Jin, et al., “Deep Learning-Based Crack Damage Detection Using Convolutional Neural Networks,” Computer- Aided Civil and Infrastructure Engineering, vol. 00, Mar. 23, 2017, pp. 1-18. [cited by applicant]
Chen, Fu-Chen, et al., “A Texture-Based Video Processing Methodology Using Bayesian Data Fusion for Autonomous Crack Detection on Metallic Surfaces,” Computer-Aided Civil and Infrastructure Engineering, vol. 0, Feb. 13,… [cited by applicant]
Chen, Fu-Chen, et al., “NB-CNN: Deep Learning-based Crack Detection Using Convolutional Neural Network and Naïve Bayes Data Fusion,” IEEE, IEEE Transactions on Industrial Electronics, Oct. 19, 2017, pp. 1-10. [cited by applicant]
Chen, Liang-Chien, et al., “Measuring System for Cracks in Concrete Using Multitemporal Images,” ASCE, American Society of Civil Engineers, Journal of Surveying Engineering, vol. 132, Issue 2, May 2006, pp. 77-82. [cited by applicant]
Cho, Yong K., et al., “3D As-Is Building Energy Modeling and Diagnostics: A Review of State-Of-The-Art,” Elsevier Ltd, Elsevier, Science Direct, Advanced Engineering Informatics, vol. 29, Apr. 11, 2015, pp. 184-195. [cited by applicant]
“ContextCapture: Software to Automatically Generate Detailed 3D Models from Photographs,” Quick Start Guide, Reality Modeling, CONNECT Edition, Bentley, Mar. 15, 2017, pp. 1-34. [cited by applicant]
Davoudi, Rouzbeh, et al., “Structural Load Estimation Using Machine Vision and Surface Crack Patterns for Shear-Critical RC Beams and Slabs,” American Society of Civil Engineers, ASCE, Journal of Computing in Civil Engi… [cited by applicant]
Dollár, Piotr, et al., “Structured Forests for Fast Edge Detection,” IEEE, 2013 IEEE International Conference on Computer Vision, Sydney, NSW, Australia, Dec. 1-8, 2013, pp. 1-8. [cited by applicant]
Dorafshan, “Automatic Surface Crack Detection in Concrete Structures Using OTSU Thresholding and Morphological Operations,” Utah State University, CEE Faculty Publications, Civil and Environmental Engineering, Paper 123… [cited by applicant]
Fan, Zhun, et al., “Automatic Pavement Crack Detection Based on Structured Prediction with the Convolutional Neural Network,” arXiv, Cs.CV, Feb. 1, 2018, pp. 1-9. [cited by applicant]
Fathi, Habib, et al., “Automated As-built 3D Reconstruction of Civil Infrastructure Using Computer Vision: Achievements, Opportunities, and Challenges,” Elsevier Ltd, Elsevier, Science Direct, Advanced Engineering Infor… [cited by applicant]
Fukushima, Kunihiko, “Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position,” Springer-Verlag, Biological Cybernetics, vol. 36, Apr. 1980, pp. 193-20… [cited by applicant]
German, Stephanie, et al., “Machine Vision-Enhanced Postearthquake Inspection,” ASCE, American Society of Civil Engineers, Journal of Computing in Civil Engineering, vol. 27, Issue 6, Nov./Dec. 2013, pp. 622-634. [cited by applicant]
Gulgec, Nur Sila, et al., “Convolutional Neural Network Approach for Robust Structural Damage Detection and Localization,” American Society of Civil Engineers, ASCE, Journal of Computing in Civil Engineering, vol. 33, I… [cited by applicant]
He, Kaiming, et al., “Deep Residual Learning from Image Recognition,” CVF (Computer Vision Foundation), CVPR, IEEE, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVRP), Las Vegas, NV, USA, Jun. 27-30,… [cited by applicant]
Hengmeechai, Jantira, “Automated Analysis of Sewer Inspection Closed Circuit Television Videos Using Image Processing Techniques,” Thesis, Faculty of Graduate Studies and Research, University of Regina, Regina, Saskatch… [cited by applicant]
Jahanshahi, Mohammad R., et al., “Adaptive Vision-based Crack Detection Using 3D Scene Reconstruction for Condition Assessment of Structures,” Elsevier B.V., Elsevier, SciVerse Science Direct, Automation in Construction… [cited by applicant]
Jahanshahi, Mohammad R., et al., “A New Methodology for Non-Contact Accurate Crack Width Measurement Through Photogrammetry for Automated Structural Safety Evaluation,” IOP Publishing, Smart Materials and Structures, vo… [cited by applicant]
Kim, Byunghyun, et al., “Automated Vision-Based Detection of Cracks on Concrete Surfaces Using a Deep Learning Technique,” MDPI, Sensors, vol. 18, Oct. 14, 2018, pp. 1-18. [cited by applicant]
Koch, Christian, et al., “A Review on Computer Vision Based Defect Detection and Condition Assessment of Concrete and Asphalt Civil Infrastructure,” Elsevier Ltd, Elsevier, Science Direct, Advanced Engineering Informati… [cited by applicant]
“Kuraves-Actis: Auto Crack Tracing Image System,” Nexco West, Nexco West USA Inc., May 2018, pp. 1-2. [cited by applicant]
Mohan, Arun, et al., “Crack Detection Using Image Processing: A Critical Review and Analysis,” Elsevier B.V., Faculty of Engineering, Alexandria University, Alexandria Engineering Journal, Jan. 2017, pp. 1-12. [cited by applicant]
Moon, Hyeong-Gyeong, et al., “Intelligent Crack Detecting Algorithm on the Concrete Crack Image Using Neural Network,” In Proceedings of the 28th International Symposium on Automation and Robotics in Construction, ISARC… [cited by applicant]
Neogi, Nirbhar, et al., “Review of Vision-Based Steel Surface Inspection Systems,” Springer, EURASIP Journal on Image and Video Processing, vol. 50, Nov. 13, 2014, pp. 1-19. [cited by applicant]
Park, Somin, et al., “Patch-Based Crack Detection in Black Box Images Using Convolutional Neural Networks,” American Society of Civil Engineers, ASCE, Journal of Computing in Civil Engineering, vol. 33, Issue 3, May 201… [cited by applicant]
Påatråucean, Viorica, et al., “State of Research in Automatic As-built Modelling,” Elsevier Ltd, Elsevier, Science Direct, Advanced Engineering Informatics, vol. 29, Feb. 3, 2015, pp. 162-171. [cited by applicant]
Pereira, Fábio Celestino, et al., “Embedded Image Processing Systems for Automatic Recognition of Cracks Using UAVs,” Elsevier Ltd, IFAC (International Federation of Automatic Control), Elsevier, ScienceDirect, IFAC Pap… [cited by applicant]
Rabah, Mostafa, et al., “Automatic Concrete Cracks Detection and Mapping of Terrestrial Laser Scan Data,” Elsevier B.V., National Research Institute of Astronomy and Geophysics, NRIAG Journal of Astronomy and Geophysics… [cited by applicant]
Sarshar, Nima, et al., “Video Processing Techniques for Assisted CCTV Inspection and Condition Rating of Sewers,” CHI Journal, Journal of Water Management Modeling, R2350-08, Jan. 2009, pp. 129-147. [cited by applicant]
Seo, JoonOh, et al., “Computer Vision Techniques for Construction Safety and Health,” Elsevier Ltd, Elsevier, Science Direct, Advanced Engineering Informatics, vol. 29, Feb. 21, 2015, pp. 239-251. [cited by applicant]
Sinha, Sunil K., et al., “Automated Detection of Cracks in Buried Concrete Pipe Images,” Elsevier B.V., Elsevier, Automation in Construction, vol. 15, Issue 1, Jan. 2006, pp. 58-72. [cited by applicant]
Sollazzo, G., et al., “Hybrid Procedure for Automated Detection of Cracking with 3D Pavement Data,” American Society of Civil Engineers, ASCE, Journal of Computing in Civil Engineering, vol. 30, Issue 6, Nov. 2016, pp. … [cited by applicant]
Son, Hyojoo, et al., “As-built Data Acquisition and its Use in Production Monitoring and Automated Layout of Civil Infrastructure: A Survey,” Elsevier Ltd, Elsevier, Science Direct, Advanced Engineering Informatics, vol… [cited by applicant]
Teizer, Jochen, “Status Quo and Open Challenges in Vision-Based and Tracking Resources on Infrastructure Construction Sites,” Elsevier Ltd, Elsevier, Science Direct, Advanced Engineering Informatics, Apr. 20, 2015, pp. … [cited by applicant]
Torok, Matthew, et al., “Image-Based Automated 3D Crack Detection for Post-disaster Building Assessment,” American Society of Civil Engineers, ASCE, Journal of Computing in Civil Engineering, vol. 28, Issue 5, Sep. 2014… [cited by applicant]
U.S. Appl. No. 16/796,462, filed Feb. 20, 2020 by Zheng Yi Wu, et al. for Evolutionary Deep Learning With Extended Kalman Filter for Modeling and Data Assimilation, 1-40. [cited by applicant]
U.S. Appl. No. 16/931,785, filed Jul. 17, 2020 by Zheng Yi Wu, et al. for Finite Element Calibration for Structural Load Identification, 1-28. [cited by applicant]
Wang, Ruisheng, “3D Building Modeling Using Images and LiDAR: A Review,” Taylor & Francis Group, International Journal of Image and Data Fusion, vol. 4, No. 4, Jul. 5, 2013, pp. 273-292. [cited by applicant]
Wu, Liuliu, et al., “Improvement of Crack-Detection Accuracy Using a Novel Crack Defragmentation Technique in Image-Based Road Assessment,” American Society of Civil Engineers, ASCE, Journal of Computing in Civil Engine… [cited by applicant]
Yamaguchi, Tomoyuki, et al., “Image-Based Crack Detection for Real Concrete Surfaces,” Paper, Wiley InterScience, Institute of Electrical Engineers of Japan, John Wiley & Sons, Inc., Transactions on Electrical and Elect… [cited by applicant]
Yang, Jun, et al., “Construction Performance Monitoring Via Still Images, Time-lapse Photos, and Video Streams: Now, Tomorrow, and the Future,” Elsevier Ltd, Elsevier, Science Direct, Advanced Engineering Informatics, v… [cited by applicant]
Yeum, C. M., et al., “Autonomous Image Localization for Visual Inspection of Civil Infrastructure,” IOP Publishing Ltd., Smart Materials and Structures, vol. 26, Feb. 21, 2017, pp. 1-12. [cited by applicant]
Yeum, Chul Min, et al., “Vision-Based automated Crack Detection for Bridge Inspection,” Computer-Aided Civil and Infrastructure Engineering, vol. 30, May 15, 2015, pp. 759-770. [cited by applicant]
Yokoyama, Suguru, et al., “Development of an Automatic Detector of Cracks in Concrete Using Machine Learning,” Elsevier Ltd, The Authors, Elsevier, Science Direct, Procedia Engineering, vol. 171, Dec. 2017, pp. 1250-125… [cited by applicant]
Zhang, Kaige, et al., “Unified Approach to Pavement Crack and Sealed Crack Detection Using Preclassification Based on Transfer Learning,” American Society of Civil Engineers, ASCE, Journal of Computing in Civil Engineer… [cited by applicant]
Zhang, Lei, et al., “Road Crack Detection Using Deep Convolutional Neural Network,” IEEE, 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, Sep. 25-28, 2016, pp. 1-5. [cited by applicant]