IP Library › Granted Patent US 12,217,408
Granted Patent B2
US 12,217,408 · App. 17/572,806 · Granted Feb 4, 2025

Semantic deep learning and rule optimization for surface corrosion detection and evaluation

Inventors: Zheng Yi Wu (Watertown, CT); Atiqur Rahman (Fremont, CA); Rony Kalfarisi (Singapore, SG)
Assignee: Bentley Systems, Incorporated
G06T7/0004G06T7/11G06T2200/24G06T2207/10024G06T2207/20081G06T2207/20084G06T2207/20092G06T2207/30136
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,217,408
App. No.
17/572,806
Granted
Feb 4, 2025
Kind
B2
Abstract

In various example embodiments, techniques are provided for training and/or using a semantic deep learning model, such as a segmentation-enabled CNN model, to detect corrosion and enable its quantitative evaluation. An application may include a training dataset generation tool capable of semi-automatic generation of a training dataset that includes images with labeled corrosion segments. The application may use the labeled training dataset to train a semantic deep learning model to detect and segment corrosion in images of an input dataset at the pixel-level. The application may apply an input dataset to the trained semantic deep learning model to produce a semantically segmented output dataset that includes labeled corrosion segments. The application may include an evaluation tool that quantitatively evaluates corrosion in the semantically segmented output dataset, to allow severity of the corrosion to be classified.

Claims (52)

1. A method for training and/or using a semantic deep learning model to detect surface corrosion, comprising:

receiving, by a training dataset generation tool of an application executing on one or more computing devices, a training dataset that includes a plurality of images of infrastructure having at least some surface corrosion;

applying, by the training dataset generation tool, an unsupervised image segmentation algorithm to segment a first portion of the images of the training dataset;

prompting a user to manually label at least some of the segments of the first portion as corrosion segments to produce labeled images;

optimizing, by the training dataset generation tool, classification rules based on the labeled images of the first portion to produce a rule-based classifier;

applying, by the training dataset generation tool, the rule-based classifier to a second portion of the images of the training dataset to automatically segment and label at least some of the segments of the second portion as corrosion segments; and

using the labeled training dataset to train the semantic deep learning model to detect and segment corrosion in images of an input dataset.

2. The method of claim 1 , wherein the first portion includes a smaller number of images than the second portion.

3. The method of claim 1 , wherein the unsupervised image segmentation algorithm is a factor-based texture segmentation (FSEG) algorithm.

4. The method of claim 1 , wherein the plurality of images are red-green-blue (RGB) images, the classification rules are RGB color channel-based classification rules and the rule-based classifier is a RGB color channel-based classifier.

5. The method of claim 1 , wherein the semantic deep learning model is a semantic segmentation-enabled convolutional neural network (CNN) model.

6. The method of claim 1 , wherein the using the labeled training dataset to train the semantic deep learning model comprises:

minimizing a total loss function of the semantic deep learning model that weights together misclassification of corrosion pixels and misclassification of non-corrosion pixels.

7. The method of claim 1 , further comprising:

applying an input dataset that includes a plurality of images of infrastructure to the trained semantic deep learning model to produce a semantically segmented output dataset, the semantically segmented output dataset including labeled corrosion segments.

8. The method of claim 7 , further comprising:

displaying, by the application, in a user interface or storing to memory/storage of the one or more computing devices, indications of the corrosion segments.

9. The method of claim 7 , further comprising:

calculating a degree of corrosion index for each corrosion segment based on a mean grayscale value of pixels within the corrosion segment's area.

10. The method of claim 9 , further comprising:

dividing, by the application, corrosion segments into categories based on a comparison of the degree of corrosion index of each corrosion segment to one or more thresholds.

11. The method of claim 10 , further comprising:

displaying, by the application, in a user interface or storing to memory/storage of the one or more computing devices, at least one of the degree of corrosion index or the category of each corrosion segment.

12. A method for training and/or using a semantic deep learning model to detect surface corrosion, comprising:

training, by an application executing on one or more computing devices, the semantic deep learning model to detect and segment corrosion in images of an input dataset;

applying an input dataset that includes a plurality of images of infrastructure to the trained semantic deep learning model to produce a semantically segmented output dataset, the semantically segmented output dataset including labeled corrosion segments;

calculating a degree of corrosion index for each corrosion segment based on a mean grayscale value of pixels within the corrosion segment's area; and

displaying, by the application, in a user interface or storing to memory/storage of the one or more computing devices, the degree of corrosion index of each corrosion segment.

13. The method of claim 12 , further comprising:

dividing, by the application, corrosion segments into categories based on a comparison of the degree of corrosion index of each corrosion segment to one or more thresholds.

14. The method of claim 13 , further comprising:

displaying, by the application, in a user interface or storing to memory/storage of the one or more computing devices the category of each corrosion segment.

15. The method of claim 12 , wherein the training further comprises:

receiving, by a training dataset generation tool of the application, a training dataset that includes a plurality of images of infrastructure having at least some surface corrosion;

applying, by the training dataset generation tool, an unsupervised image segmentation algorithm to segment a first portion of the images of the training dataset;

prompting a user to manually label at least some of the segments of the first portion as corrosion segments to produce labeled images;

optimizing, by the training dataset generation tool, classification rules based on the labeled images of the first portion to produce a rule-based classifier;

applying, by the training dataset generation tool, the rule-based classifier to a second portion of the images of the training dataset to automatically segment and label at least some of the segments of the second portion as corrosion segments; and

using the labeled training dataset to train the semantic deep learning model.

16. The method of claim 15 , wherein the first portion includes a smaller number of images than the second portion.

17. The method of claim 15 , wherein the unsupervised image segmentation algorithm is a factor-based texture segmentation (FSEG) algorithm, the plurality of images are red-green-blue (RGB) images, the classification rules are RGB color channel-based classification rules, the rule-based classifier is a RGB color channel-based classifier, and the semantic deep learning model is a semantic segmentation-enabled convolutional neural network (CNN) model.

18. A non-transitory electronic device readable medium having instructions stored thereon that when executed on one or more processors of one or more electronic devices are operable to:

receive a training dataset that includes a plurality of images of infrastructure having at least some surface corrosion;

apply an image segmentation algorithm to segment a first portion of the images of the training dataset;

receive user-provided labels for at least some of the segments of the first portion to produce labeled images;

optimize classification rules based on the labeled images of the first portion to produce a rule-based classifier;

apply the rule-based classifier to a second portion of the images of the training dataset to automatically segment and label at least some of the segments of the second portion; and

use the labeled training dataset to train a semantic deep learning model to detect and segment corrosion in images of an input dataset.

19. The non-transitory electronic device readable medium of claim 18 , further comprising:

applying an input dataset that includes a plurality of images of infrastructure to the trained semantic deep learning model to produce a semantically segmented output dataset, the semantically segmented output dataset including labeled corrosion segments.

20. The non-transitory electronic device readable medium of claim 19 , further comprising:

calculating a degree of corrosion index for each corrosion segment based on a mean grayscale value of pixels within the corrosion segment.

Assignments (2)
SECURITY INTEREST Recorded Oct 25, 2024
From: BENTLEY SYSTEMS, INCORPORATED
To: PNC BANK, NATIONAL ASSOCIATION
Reel/Frame 069268/0042 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 11, 2022
From: WU, ZHENG YI; RAHMAN, ATIQUR; KALFARISI, RONY
To: BENTLEY SYSTEMS, INCORPORATED
Reel/Frame 058617/0211 →
Continuity (1)
Related Publication 20230222643A1 · Jul 13, 2023
References Cited (58)
US 11600061B1 · Jeong · 2023 [cited by examiner]
US 20200175324A1 · Takahashi · 2020 [cited by examiner]
US 20200175352A1 · Cha · 2020 [cited by examiner]
US 20200355601A1 · Amer · 2020 [cited by examiner]
US 20220036132A1 · Oruganty · 2022 [cited by examiner]
US 20220058809A1 · Fuchs · 2022 [cited by examiner]
US 20220262104A1 · Salman · 2022 [cited by examiner]
US 20230003663A1 · Horita · 2023 [cited by examiner]
US 20230003687A1 · Vaganay · 2023 [cited by examiner]
Abdelrazig, Yassir Abdalla, “Construction Quality assessment: A Hybrid Decision Support Model Using Image Processing and Neural Learning for Intelligent Defects Recognition,” Thesis, Graduate School, Purdue University, … [cited by applicant]
Alkanhal, Tawfeeq A., “Image Processing Techniques Applied for Pitting Corrosion Analysis,” IJRET: International Journal of Research in Engineering and Technology, vol. 3, Issue 1, Jan. 2014, pp. 385-391. [cited by applicant]
Atha, Deegan, et al., “Evaluation of Deep Learning Approaches Based on Convolutional Neural Networks for Corrosion Detection,” Structural Health Monitoring, SHM, vol. 17, No. 5, Nov. 6, 2017, pp. 1110-1128. [cited by applicant]
Badrinarayanan, Vijay, et al., “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation,” IEEE, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, No. 12, Dec. 2017, pp. 24… [cited by applicant]
Bonnin-Pascual, Francisco, et al., “Corrosion Detection for Automated Visual Inspection,” InTechOpen, Chapter, Feb. 2014, pp. 1-13. [cited by applicant]
Castrejón, Lluis, et al., “Annotating Object Instances with a Polygon-RNN,” arXiv, arXiv:1704.05548v1 [cs.CV], Apr. 18, 2017, pp. 1-9. [cited by applicant]
Chen, Liang-Chieh, et al., “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs,” IEEE, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40… [cited by applicant]
Chen, Po-Han, et al., “Artificial Intelligence Application to Bridge Painting Assessment,” Elsevier Science B.V., Elsevier, Automation in Construction, vol. 12, Issue 4, Jul. 2003, pp. 431-445. [cited by applicant]
Chen, Po-Han, et al., “Automated Bridge Coating Defect Recognition Using Adaptive Ellipse Approach,” Elsevier B.V., Elsevier, Science Direct, Automation in Construction, vol. 18, Issue 5, Aug. 2009, pp. 632-643. [cited by applicant]
Chen, Xianjie, et al., “Detect What You Can: Detecting and Representing Objects Using Holistic Models and Body Parts,” arXiv, arXiv:1406.2031v1 [cs.CV], Jun. 8, 2014, pp. 1-8. [cited by applicant]
Choi, K. Y., et al., “Morphological Analysis and Classification of Types of Surface Corrosion Damage by Digital Image Processing,” Elsevier Ltd., Elsevier, Science Direct, Corrosion Science, vol. 47, Jul. 2, 2004, pp. 1… [cited by applicant]
Cordts, Marius, et al., “The Cityscapes Dataset for Semantic Urban Scene Understanding,” IEEE, IEEE Computer Society, 2016 IEEE Conference on Computer Vision Pattern Recognition (CVPR), Las Vegas, NV, USA, Jun. 27-30, 2… [cited by applicant]
Duy, Le Dinh, et al., “Deep Learning in Semantic Segmentation of Rust in Images,” ACM, Inc., ACM, ICSCA 2020: Proceedings of the 2020 9th International Conference on Software and Computer Applications, Feb. 2020, pp. 1-… [cited by applicant]
Enikeev, Marat, et al., “Analysis of Corrosion Process Development on Metals by Means of Computer Vision,” Engineering Journal, vol. 21, Issue 4, Jul. 31, 2017, pp. 183-192. [cited by applicant]
Everingham, Mark, et al., “The PASCAL Visual Object Classes Challenge: A Retrospective,” Springer Science+Business Media New York, Springer, International Journal of Computer Vision, vol. 111, Jun. 25, 2014, pp. 98-136. [cited by applicant]
Farabet, Clement, et al., “Learning Hierarchical Features for Scene Labeling,” IEEE, IEEE Computer Society, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, No. 8, Aug. 2013, pp. 1915-1929. [cited by applicant]
Forkan, Abdur Rahim Mohammad, et al., “CorrDetector: A Framework for Structural Corrosion Detection from Drone Images Using Ensemble Deep Learning,” arXiv, arXiv:2102.04686v1 [cs.CV], Feb. 9, 2021, pp. 1-26. [cited by applicant]
Fulkerson, Brian, et al., “Class Segmentation and Object Localization with Superpixel Neighborhoods,” IEEE, 2009 IEEE 12 [cited by applicant]
Furuta, Hitoshi, et al., “Neural Network Analysis of Structural Damage Due to Corrosion,” IEEE, Proceedings of 3rd International Symposium on Uncertainty Modeling and Analysis and Annual Conference of the North American… [cited by applicant]
Ghanta, Sindhu, et al., “Wavelet Domain Detection of Rust in Steel Bridge Images,” IEEE, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Prague, Czech Republic, May 22-27, 2011, p… [cited by applicant]
He, Kaiming, et al., “Mask R-CNN,” IEEE, IEEE Computer Society, 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, Oct. 22-29, 2017, pp. 2980-2988. [cited by applicant]
Hoskere, V., et al., “Vision-Based Structural Inspection Using Multiscale Deep Convolutional Neural Networks,” 3 [cited by applicant]
“InspectTech: Inspection, Maintenance, and Asset Management for Transportation Infrastructure: Bridge, Tunnel, Road, Rail, and Transit,” Bentley, Bentley Systems, Incorporated, Product Data Sheet, Nov. 2014, pp. 1-2. [cited by applicant]
Jahanshahi, Mohammad R., “Parametric Performance Evaluation of Wavelet-Based Corrosion Detection Algorithms for Condition Assessment of Civil Infrastructure Systems,” American Society of Civil Engineers, ASCE, Journal o… [cited by applicant]
Kalfarisi, Rony, et al., “Crack Detection and Segmentation Using Deep Learning with 3D Mesh Model for Quantitative Assessment and Integrated Visualization,” American Society of Civil Engineers, ASCE, Journal of Computin… [cited by applicant]
Koch, Gerhardus H., et al., “Corrosion Costs and Preventive Strategies in the United States,” NACE International, CC Technologies Laboratories, Incorporated, Federal Highway Administration, Publication No. FHWA-RD-01-15… [cited by applicant]
Lecun, Yann, et al., “Gradient-Based learning Applied to Document Recognition,” IEEE, Proceedings of the IEEE, vol. 86, No. 11, Nov. 1998, pp. 2278-2324. [cited by applicant]
Lee, S., et al., “Performance Comparison of Bridge Coating Defect Recognition Methods,” NACE International, Corrosion, Corrosion Science Section, Jan. 2005, pp. 12-20. [cited by applicant]
Lee, Sangwook, et al., “Automated Recognition of Surface Defects Using Digital Color Image Processing,” Elsevier B.V., Elsevier, Automation in Construction, vol. 15, Issue 4, Jul. 2006, pp. 540-549. [cited by applicant]
Lim, Hyung Jin, et al., “Steel Bridge Corrosion Inspection with Combined Vision and Thermographic,” Structural Health Monitoring, SHM, vol. 20, No. 6, Feb. 11, 2021, pp. 3424-3435. [cited by applicant]
Lin, Tsung-Yi, et al., “Feature Pyramid Networks for Object Detection,” CVF, Computer Vision Foundation, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, Jul. 21-2… [cited by applicant]
Ling, Huan, et al., “Fast Interactive Object Annotation with Curve-GCN,” CVF, Computer Vision Foundation, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, Ju… [cited by applicant]
Liu, Wei, et al., “ParseNet: Looking Wider to See Better,” arXiv, arXiv:1506.04579v2 [cs.CV], Conference Paper, Nov. 19, 2015, pp. 1-11. [cited by applicant]
Maninis, K.-K., et al., “Deep Extreme Cut: From Extreme Points to Object Segmentation,” IEEE, IEEE Computer Society, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, Jun. 18-… [cited by applicant]
Mottaghi, Roozbeh, et al., “The Role of Context for Object Detection and Semantic Segmentation in the Wild,” IEEE, IEEE Computer Society, 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, US… [cited by applicant]
Nash, Will, et al., “Quantity Beats Quality for Semantic Segmentation of Corrosion in Images,” arXiv, arXiv:1807.03138v1 [cs.CV], Jun. 30, 2018, pp. 1-10. [cited by applicant]
Noh, Hyeonwoo, et al., “Learning Deconvolution Network for Semantic Segmentation,” arXiv, arXiv:1505.04366v1 [cs.CV], May 17, 2015, pp. 1-10. [cited by applicant]
Petricca, Luca, et al., “Corrosion Detection Using A.I.: A Comparison of Standard Computer Vision Techniques and Deep Learning Model,” Proceedings of the Sixth International Conference on Computer Science, Engineering a… [cited by applicant]
Pidaparti, Ramana M., et al., “Classification of Corrosion Defects in NiAl Bronze Through Image Analysis,” Elsevier Ltd., Elsevier, Science Direct, Corrosion Science, vol. 52, Jul. 13, 2010, pp. 3661-3666. [cited by applicant]
Ronneberger, Olaf, et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation,” arXiv, arXiv:1505.04597v1 [cs.CV], May 18, 2015, pp. 1-8. [cited by applicant]
Russakovsky, Olga, et al., “ImageNet Large Scale Visual Recognition Challenge,” arXiv, arXiv: 1409.0575v3 [cs.CV], Jan. 30, 2015, pp. 1-43. [cited by applicant]
Simonyan, Karen, et al., “Very Deep Convolutional Networks for Large-Scale Image Recognition,” arXiv, arXiv:1409.1556v6 [cs.CV], Conference Paper, ICLR 2015, Apr. 10, 2015, pp. 1-14. [cited by applicant]
Shotton, Jamie, et al., “Semantic Texton Forests for Image Categorization and Segmentation,” IEEE, 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA, Jun. 23-28, 2008, pp. 1-8. [cited by applicant]
Shotton, Jamie, et al., “TextonBoost for Image Understanding: Multi-Class Object Recognition and Segmentation by Jointly Modeling Texture, Layout, and Context,” Jul. 2, 2007, pp. 1-30. [cited by applicant]
Tu, Zhuowen, et al., “Auto-Context and Its Application to High-Level Vision Tasks and 3D Brain Image Segmentation,” IEEE, IEEE Computer Society, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, N… [cited by applicant]
Wu, Z. Y., et al., “Darwin Optimization User Manual,” Darwin Optimization, Applied Research, Bentley, Bentley Systems, Incorporated, Sep. 12, 2012, pp. 1-20. [cited by applicant]
Yuan, Jiangye, et al., “Factorization-Based Texture Segmentation,” IEEE, IEEE Transactions on Image Processing, vol. 24, No. 11, Nov. 2015, pp. 3488-3497. [cited by applicant]
Zeiler, Matthew, et al., “Visualizing and Understanding Convolutional Networks,” Springer International Publishing Switzerland, 13th European Conference, Zurich, Switzerland, Proceedings, Part I, European Conference on … [cited by applicant]
Zhao, Hengshuang, et al., “Pyramid Scene Parsing Network,” CVF, Computer Vision Foundation, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, Jul. 21-26, 2017, pp. … [cited by applicant]