IP Library › Granted Patent US 12,475,375
Granted Patent B2
US 12,475,375 · App. 17/769,304 · Granted Nov 18, 2025

Systems and methods for assessing infrastructure

Inventors: Miguel Dickson (Pittsburgh, PA); Benjamin T. Schmidt (Penn Hills, PA)
Assignee: Michelin Mobility Intelligence, Inc.
G06N3/082E01C23/01G06T7/0002G06V10/70G06V10/82G06V20/10G06V20/56G06T2207/10016G06T2207/20081G06T2207/30184
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,475,375
App. No.
17/769,304
Granted
Nov 18, 2025
Kind
B2
Abstract

Systems and methods for assessing infrastructure assets. In some embodiments, sensor data collected from a plurality of infrastructure assets may be analyzed, and an infrastructure asset may be identified as a candidate for a selected type of maintenance. The infrastructure asset may include a road segment, and the sensor data may include an image of a pavement surface along the road segment. One or more machine learning models may be used to identify a portion of the image exhibiting a selected type of damage, such as cracking, chipping, pothole, etc.

Claims (107)

1 . A system comprising:

at least one processor; and

at least one non-transitory computer-readable medium having encoded thereon instructions which, when executed, program the at least one processor to:

determine, based on sensor data collected from a first plurality of road segments, an assessment for each road segment of the first plurality of road segments, the assessment being indicative of an overall condition of the road segment;

use one or more selected thresholds to filter the first plurality of road segments, based on the respective assessments, thereby obtaining a second plurality of road segments, wherein:

the at least one processor is programmed to filter out road segments having assessments above a selected threshold indicative of severe damage;

identify, based on the sensor data, at least one road segment from the second plurality of road segments as a candidate for a selected type of maintenance, wherein:

the sensor data comprises an image of the at least one road segment; and

the at least one processor is programmed to use at least one machine learning model to identify, from the image, one or more instances of physical damage exhibited by the at least one road segment; and

construct a candidate batch comprising road segments that are identified as candidates for the selected type of maintenance.

2 . The system of claim 1 , wherein:

the one or more instances of physical damage comprise one or more instances of transverse or longitudinal cracking; and

the selected type of maintenance comprises crack sealing.

3 . The system of claim 1 , wherein:

the at least one machine learning model is configured to determine whether the selected type of maintenance has been performed on the at least one road segment; and

the at least one road segment is identified as a candidate for the selected type of maintenance in response to determining that the selected type of maintenance has not been performed on the at least one road segment.

4 . The system of claim 1 , wherein:

the at least one machine learning model is configured to determine whether the at least one road segment exhibits an insufficient number of instances of physical damage that is effectively addressable by the selected type of maintenance; and

the at least one road segment is identified as a candidate for the selected type of maintenance in response to determining that the at least one road segment does not exhibit an insufficient number of instances of physical damage that is effectively addressable by the selected type of maintenance.

5 . The system of claim 1 , wherein:

the at least one machine learning model is configured to determine whether the at least one road segment exhibits an excessive number of instances of physical damage of one or more types other than a selected type of physical damage.

6 . The system of claim 1 , wherein:

the at least one processor is programmed to construct a candidate batch at least in part by prioritizing a plurality of candidates for the selected type of maintenance, over other candidates for the selected type of maintenance, based on geospatial proximity of the plurality of candidates.

7 . The system of claim 1 , wherein;

the sensor data comprises at least one image of a selected road segment;

the at least one processor is programmed to determine the assessment of the selected road segment at least in part by applying at least one first machine learning model to the at least one image to obtain a machine assessment of the selected road segment; and

the at least one processor is further programmed to determine whether the machine assessment of the selected road segment is reliable, at least in part by:

determining whether there is agreement among a plurality of human assessments of the selected road segment, wherein the plurality of human assessments are produced, respectively, by a plurality of human reviewers reviewing the selected road segment; and

in response to determining that there is agreement among the plurality of human assessments of the selected road segment, using the plurality of human assessments of the selected road segment to improve correctness of the machine assessment of the selected road segment.

8 . The system of claim 7 , wherein:

the sensor data comprises a sequence of images acquired by a moving camera;

the at least one image of the selected road segment comprises a first image in the sequence of images;

the at least one processor is further programmed to use at least one second machine learning model to generate a predicted human assessment of the first image;

the at least one second machine learning model is configured to receive, as input, at least one actual human assessment of at least one second image in the sequence of images; and

the at least one second image comprises an image that, in the sequence of images, immediately precedes or immediately follows the first image.

9 . The system of claim 8 , wherein:

the at least one second machine learning model is further configured to receive, as input, a machine assessment of the first image and/or at least one machine assessment of the at least one second image.

10 . The system of claim 7 , wherein the at least one processor is further programmed to:

determine, for each human reviewer of the plurality of human reviewers, a per-reviewer central tendency value for a plurality of human assessments produced by that human reviewer reviewing a plurality of images; and

determine, for a given human reviewer, whether that human reviewer's per-reviewer central tendency value is within a selected range of a central tendency value of the per-reviewer central tendency values of the plurality of human reviewers.

11 . The system of claim 7 , wherein the at least one processor is further programmed to:

assess a human reviewer based on at least one frequency selected from a group consisting of:

a frequency at which the human reviewer is in a unanimous judgment;

a frequency at which the human reviewer is part of a majority in a majority judgment;

a frequency at which the human reviewer is part of a minority in a majority judgment; and

a frequency at which the human reviewer is in a split judgment.

12 . The system of claim 7 , wherein:

the sensor data comprises a plurality of images of the selected road segment; and

the at least one processor is further programmed to:

determine, for each image of the plurality of images of the selected road segment, a per-image dispersion value for a plurality of human assessments of the selected road segment produced, respectively, by the plurality of human reviewers reviewing that image of the plurality of images of the selected road segment; and

determine, based on the per-image dispersion values associated, respectively, with the plurality of images of the selected road segment, an aggregate degree of agreement for the selected road segment among the plurality of human reviewers.

13 . A computer-implemented method comprising acts of:

determining, based on sensor data collected from a first plurality of road segments, an assessment for each road segment of the first plurality of road segments, the assessment being indicative of an overall condition of the road segment;

using one or more selected thresholds to filter the first plurality of road segments, based on the respective assessments, thereby obtaining a second plurality of road segments, wherein:

using one or more selected thresholds to filter the first plurality of road segments comprises filtering out road segments having assessments above a selected threshold indicative of severe damage;

identifying, based on the sensor data, at least one road segment from the second plurality of road segments as a candidate for a selected type of maintenance, wherein:

the sensor data comprises an image of the at least one road segment; and

identifying the at least one road segment as a candidate for the selected type of maintenance comprises using at least one machine learning model to identify, from the image, one or more instances of physical damage exhibited by the at least one road segment; and

constructing a candidate batch comprising road segments that are identified as candidates for the selected type of maintenance.

14 . The computer-implemented method of claim 13 , wherein:

the one or more instances of physical damage comprise one or more instances of transverse or longitudinal cracking; and

the selected type of maintenance comprises crack sealing.

15 . The computer-implemented method of claim 13 , wherein:

the at least one machine learning model is configured to determine whether the selected type of maintenance has been performed on the at least one road segment; and

the at least one road segment is identified as a candidate for the selected type of maintenance in response to determining that the selected type of maintenance has not been performed on the at least one road segment.

16 . The computer-implemented method of claim 13 , wherein:

the at least one machine learning model is configured to determine whether the at least one road segment exhibits an insufficient number of instances of physical damage that is effectively addressable by the selected type of maintenance; and

the at least one road segment is identified as a candidate for the selected type of maintenance in response to determining that the at least road segment does not exhibit an insufficient number of instances of physical damage that is effectively addressable by the selected type of maintenance.

17 . The computer-implemented method of claim 13 , wherein:

the at least one machine learning model is configured to determine whether the at least one road segment exhibits an excessive number of instances of damage of one or more types other than a selected type of physical damage.

18 . The computer-implemented method of claim 13 , wherein:

the act of constructing a candidate batch comprises prioritizing a plurality of candidates for the selected type of maintenance, over other candidates for the selected type of maintenance, based on geospatial proximity of the plurality of candidates.

19 . The computer implemented method of claim 13 , wherein:

the sensor data comprises at least one image of a selected road segment;

the act of determining the assessment of the selected road segment comprises applying at least one first machine learning model to the at least one image to obtain a machine assessment of the selected road segment; and

the method further comprises an act of determining whether the machine assessment of the selected road segment is reliable, at least in part by:

determining whether there is agreement among a plurality of human assessments of the selected road segment, wherein the plurality of human assessments are produced, respectively, by a plurality of human reviewers reviewing the selected road segment; and

in response to determining that there is agreement among the plurality of human assessments of the selected road segment, using the plurality of human assessments of the selected road segment to improve correctness of the machine assessment of the selected road segment.

20 . The computer implemented method of claim 19 , wherein:

the sensor data comprises a sequence of images acquired by a moving camera;

the at least one image of the selected road segment comprises a first image in the sequence of images;

the method further comprises an act of using at least one second machine learning model to generate a predicted human assessment of the first image;

the at least one second machine learning model is configured to receive, as input, at least one actual human assessment of at least one second image in the sequence of images; and

the at least one second image comprises an image that, in the sequence of images, immediately precedes or immediately follows the first image.

21 . The computer implemented method of claim 20 , wherein: the at least one second machine learning model is further configured to receive, as input, a machine assessment of the first image and/or at least one machine assessment of the at least one second image.

22 . The computer implemented method of claim 19 , further comprising acts of:

determining, for each human reviewer of the plurality of human reviewers, a per-reviewer central tendency value for a plurality of human assessments produced by that human reviewer reviewing a plurality of images; and

determining, for a given human reviewer, whether that human reviewer's per-reviewer central tendency value is within a selected range of a central tendency value of the per-reviewer central tendency values of the plurality of human reviewers.

23 . The computer implemented method of claim 19 , further comprising an act of:

assessing a human reviewer based on at least one frequency selected from a group consisting of:

a frequency at which the human reviewer is in a unanimous judgment;

a frequency at which the human reviewer is part of a majority in a majority judgment;

a frequency at which the human reviewer is part of a minority in a majority judgment; and

a frequency at which the human reviewer is in a split judgment.

24 . The computer implemented method of claim 19 , wherein:

the sensor data comprises a plurality of images of the selected road segment; and

the method further comprises acts of:

determining, for each image of the plurality of images of the selected road segment, a per-image dispersion value for a plurality of human assessments of the selected road segment produced, respectively, by the plurality of human reviewers reviewing that image of the plurality of images of the selected road segment; and

determining, based on the per-image dispersion values associated, respectively, with the plurality of images of the selected road segment, an aggregate degree of agreement for the selected road segment among the plurality of human reviewers.

25 . At least one non-transitory computer-readable medium having encoded thereon instructions which, when executed, program at least one processor to perform a method comprising acts of:

determining, based on sensor data collected from a first plurality of road segments, an assessment for each road segment of the first plurality of road segments, the assessment being indicative of an overall condition of the road segment;

using one or more selected thresholds to filter the first plurality of road segments, based on the respective assessments, thereby obtaining a second plurality of road segments, wherein:

using one or more selected thresholds to filter the first plurality of road segments comprises filtering out road segments having assessments above a selected threshold indicative of severe damage;

identifying, based on the sensor data, at least one road segment from the second plurality of road segments as a candidate for a selected type of maintenance, wherein:

the sensor data comprises an image of the at least one road segment; and

identifying the at least one road segment as a candidate for the selected type of maintenance comprises using at least one machine learning model to identify, from the image, one or more instances of physical damage exhibited by the at least one road segment; and

constructing a candidate batch comprising road segments that are identified as candidates for the selected type of maintenance.

Assignments (2)
CHANGE OF NAME Recorded Sep 17, 2025
From: ROADBOTICS, INC.
To: MICHELIN MOBILITY INTELLIGENCE, INC.
Reel/Frame 072916/0074 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 28, 2022
From: DICKSON, MIGUEL; SCHMIDT, BENJAMIN T.
To: ROADBOTICS, INC.
Reel/Frame 060334/0969 →
Continuity (2)
Provisional Application 62915426 · Oct 15, 2019
Related Publication 20230306573A1 · Sep 28, 2023
References Cited (53)
US 6453056B2 · Laumeyer et al. · 2002 [cited by applicant]
US 6690294B1 · Zierden · 2004 [cited by applicant]
US 7697727B2 · Xu et al. · 2010 [cited by applicant]
US 9092981B2 · Suzuki et al. · 2015 [cited by applicant]
US 9196048B2 · Jahanshahi et al. · 2015 [cited by applicant]
US 9916755B1 · Ratti · 2018 [cited by applicant]
US 10147002B2 · Hartmann et al. · 2018 [cited by applicant]
US 10296794B2 · Ratti · 2019 [cited by applicant]
US 11769238B2 · Schmidt et al. · 2023 [cited by applicant]
US 20030202683A1 · Ma et al. · 2003 [cited by applicant]
US 20070127774A1 · Zhang et al. · 2007 [cited by applicant]
US 20110234749A1 · Alon · 2011 [cited by applicant]
US 20130082857A1 · Beer et al. · 2013 [cited by applicant]
US 20130216089A1 · Chen et al. · 2013 [cited by applicant]
US 20140196529A1 · Cronin et al. · 2014 [cited by applicant]
US 20160292518A1 · Banitt et al. · 2016 [cited by applicant]
US 20170032402A1 · Patsiokas et al. · 2017 [cited by applicant]
US 20190005685A1 · Mu · 2019 [cited by applicant]
US 20200334566A1 · Vianu · 2020 [cited by examiner]
US 20230196535A1 · Schmidt et al. · 2023 [cited by applicant]
US 20240127416A1 · Schmidt et al. · 2024 [cited by applicant]
CN 107092020A · 2017 [cited by applicant]
CN 109564619A · 2019 [cited by applicant]
EP 3859279A1 · 2021 [cited by applicant]
WO WO2008032305A2 · 2008 [cited by applicant]
Wu et al., “Coupling Deep Learning and UAV for Infrastructure Condition Assessment Automation”, IEEE, 2018 (Year: 2018). [cited by examiner]
Abou Charcra et al., “Fully Automated Road Defect Detection Using Street View Images”, IEEE, 14th Conference on Computer and Robot Vision, 2017 (Year: 2017). [cited by examiner]
Angulo et al., Road Damage Detection Acquisition System based on Deep Neural Networks for Physical Asset Management, Sep. 2019 (Year: 2019). [cited by examiner]
International Search Report and Written Opinion for International Application No. PCT/US2020/055508, mailed Jan. 21, 2021. [cited by applicant]
Third Party Observation for International Application No. PCT/US2020/055508, submitted Feb. 14, 2022. [cited by applicant]
International Preliminary Report on Patentability for International Application No. PCT/US2020/055508, mailed Apr. 28, 2022. [cited by applicant]
Invitation to Pay Additional Fees for International Application No. PCT/US2021/032820, mailed Jul. 15, 2021. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2021/032820, mailed Oct. 7, 2021. [cited by applicant]
[No Author Listed] Designation: D 6433-07. Standard practice for roads and parking lots pavement condition index surveys. ASTM International. https://pages.mtu.edu/˜balkire/CE5403/ASTMD6433.pdf. Jan. 2008: 1-48. [cited by applicant]
[No Author Listed] Scale-invariant feature transform. Wikipedia. https://en.wikipedia.org/wiki/Scale-invariant_feature_transform. Downloaded Mar. 12, 2020:1-19. [cited by applicant]
[No Author Listed] Structure from motion. Wikipedia. https://en.wikipedia.org/wiki/Structure_from_Motion. Downloaded Mar. 12, 2020:1-9. [cited by applicant]
Berger et al., State of the art in surface reconstruction from point clouds. Eurographics 2014 State of the Art Reports (STAR). Apr. 7, 2014;1(1):161-85. [cited by applicant]
Bruls et al., Mark yourself: Road marking segmentation via weakly-supervised annotations from multimodal data. 2018 IEEE International Conference on Robotics and Automation (ICRA). May 21-25, 2018:1863-70. [cited by applicant]
Kudinov et al., Object extraction from Mobile LiDAR point clouds with Machine Learning. Medium GeoAI. https://medium.com/geoai/object-extraction-from-mobile-lidar-point-clouds-with-machine-learning-cb15fcbb5597. Jan. 27… [cited by applicant]
Li et al., Efficient dense point cloud object reconstruction using deformation vector fields. Proceedings of the European Conference on Computer Vision (ECCV). Sep. 8-14, 2018:497-513. [cited by applicant]
Mandikal et al., 3d-psrnet: Part segmented 3d point cloud reconstruction from a single image. Proceedings of the European Conference on Computer Vision (ECCV) Workshops Sep. 8-14, 2018:1-13. [cited by applicant]
Moulon et al., Global fusion of relative motions for robust, accurate and scalable structure from motion. Proceedings of the IEEE International Conference on Computer Vision (ICCV). Dec. 1-8, 2013:3248-55. [cited by applicant]
Napier, Vision & laser for road based localisation. Robotics Research Group, Department of Engineering Science, University of Oxford. https://www.robots.ox.ac.uk/˜mobile/Theses/Napier_Thesis_shrunk.pdf. Jan. 2014:1-190. [cited by applicant]
Trigilio, Structure from motion (SfM). Humboldt State Geospatial Online GSP 216: Introduction to Remote Sensing. gsp.humboldt.edu/OLM/Courses/GSP_216_Online/lesson8-2/SfM.html. Dated 2017; Downloaded Mar. 12, 2020:1-6. [cited by applicant]
Zhou et al., Detail preserved surface reconstruction from point cloud. Sensors. Mar. 13, 2019;19(6):1278. [cited by applicant]
Sheehan et al., Continuous vehicle localisation using sparse 3D sensing, kernelised Rényi distance and fast Gauss transforms. 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Nov. 3, 2013… [cited by applicant]
Yuan et al., Robust lane detection for complicated road environment based on normal map. IEEE Access. Sep. 6, 2018;6:49679-89. [cited by applicant]
U.S. Appl. No. 17/925,958, filed Nov. 17, 2022, Schmidt et al. [cited by applicant]
PCT/US2021/032820, Sep. 18, 2022, Third Party Observation. [cited by applicant]
PCT/US2021/032820, Dec. 1, 2022, International Preliminary Report on Patentability. [cited by applicant]
Third Party Observation for International Application No. PCT/US2021/032820, submitted Sep. 18, 2022. [cited by applicant]
International Preliminary Report on Patentability for International Application No. PCT/US2021/032820, mailed Dec. 1, 2022. [cited by applicant]
Guan et al., Iterative tensor voting for pavement crack extraction using mobile laser scanning data. IEEE Transactions on Geoscience and Remote Sensing. Sep. 4, 2014;53(3):1527-37. [cited by applicant]