IP Library › Granted Patent US 12,198,423
Granted Patent B2
US 12,198,423 · App. 17/977,977 · Granted Jan 14, 2025

Method and system for assessing damage to infrastructure

Inventors: Brian N. Harvey (Bloomington, IL); Nathan L. Tofte (Downs, IL); Roger D. Schmidgall (Normal, IL); Michael Jeffrey Aviles (Bloomington, IL); Kyle Pott (Bloomington, IL); Rosemarie Geier Grant (Ellsworth, IL); Eric Haefli (Bloomington, IL); Michael Shawn Jacob (Le Roy, IL)
Assignee: STATE FARM MUTUAL AUTOMOBILE INSURANCE COMPANY
G06V20/176G06F18/24G06T7/0002G06T7/0004G06T7/0008G06T7/13G06T7/60G06T7/62G06T15/005G06V10/42G06V10/44G06V20/182G06T2200/04G06T2200/24G06T2207/10016G06T2207/10021G06T2207/10024G06T2207/10028G06T2207/10032G06T2207/10036G06T2207/30108G06T2207/30184
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,198,423
App. No.
17/977,977
Granted
Jan 14, 2025
Kind
B2
Abstract

A method and system may survey a property using aerial images captured from an unmanned aerial vehicle (UAV), a manned aerial vehicle (MAV) or from a satellite device. The method may include identifying a commercial property for a UAV to perform surveillance, and directing the UAV to hover over the commercial property and capture aerial images at predetermined time intervals. Furthermore, the method may include receiving the aerial images of the commercial property captured at the predetermined time intervals, detecting a surveillance event at the commercial property, generating a surveillance alert, and transmitting the surveillance alert to an electronic device associated with an owner of the commercial property.

Claims (52)

1. A computer-implemented method for determining a condition of a property using one or more aerial images, the method comprising:

receiving, at one or more processors, one or more aerial images of a property;

identifying, by the one or more processors, one or more portions of the one or more aerial images corresponding to one or more property components of the property;

for a particular property component:

determining, by the one or more processors, a degree of damage to the particular property component;

determining, by the one or more processors, a score indicating the degree of damage to the property component; and

determining, by the one or more processors, a condition of the particular property component based on the score;

determining, by the one or more processors, a level of risk of the property, wherein the level of risk is not based on the degree of damage of a property component;

assigning, by the one or more processors, a level of risk indicator to the property based upon the determined level of risk;

providing, by the one or more processors for display on a user interface, an indication of the condition of each property component overlaying the corresponding portion of the one or more aerial images; and

providing, by the one or more processors, the level of risk indicator for display on the user interface.

2. The computer-implemented method of claim 1 , wherein determining the condition of the particular property component includes determining, by the one or more processors, a change in condition of the particular property component over time.

3. The computer-implemented method of claim 2 , wherein determining the level of risk of the property includes determining, by the one or more processors, the level of risk of the property based upon an amount of time in which the property component is in a particular condition.

4. The computer-implemented method of claim 1 , further comprising:

determining, by the one or more processors, an insurance premium for the property based upon the level of risk.

5. The computer-implemented method of claim 1 , wherein the condition of the particular property component is categorized as at least one of: excellent, good, fair, or poor.

6. The computer-implemented method of claim 1 , wherein the one or more property components include at least one of: a window, a door, a roof, siding, an exterior wall, a garage, a backyard, a swimming pool, a lawn, a driveway, a fence, a tree, a deck, or a patio.

7. A computer system for determining a condition of a property using one or more aerial images, the system comprising:

one or more processors;

a non-transitory computer-readable memory coupled to the one or more processors, and storing thereon instructions that, when executed by the one or more processors, cause the system to:

receive one or more aerial images of a property;

identify one or more portions of the one or more aerial images corresponding to one or more property components of the property;

for a particular property component:

determine a degree of damage to the particular property component;

determine a score indicating the degree of damage to the property component; and

determine a condition of the particular property component based on the score;

determine a level of risk of the property, wherein the level of risk is not based on the degree of damage of a property component;

assign a level of risk indicator to the property based upon the determined level of risk;

provide, for display on a user interface, an indication of the condition of each property component overlaying the corresponding portion of the one or more aerial images; and

provide the level of risk indicator for display on the user interface.

8. The computer system of claim 7 , wherein the condition of the particular property component includes a change in condition of the particular property component over time.

9. The computer system of claim 8 , wherein to determine the level of risk of the property, the instructions cause the system to determine the level of risk of the property based upon an amount of time in which the property component is in a particular condition.

10. The computer system of claim 7 , wherein the instructions further cause the system to:

determine an insurance premium for the property based upon the level of risk.

11. The computer system of claim 7 , wherein the condition of the particular property component is categorized as at least one of: excellent, good, fair, or poor.

12. The computer system of claim 7 , wherein the one or more property components include at least one of: a window, a door, a roof, siding, an exterior wall, a garage, a backyard, a swimming pool, a lawn, a driveway, a fence, a tree, a deck, or a patio.

13. A non-transitory computer-readable memory coupled to one or more processors, and storing thereon instructions that, when executed by the one or more processors, cause the one or more processors to:

receive one or more aerial images of a property;

identify one or more portions of the one or more aerial images corresponding to one or more property components of the property;

for a particular property component:

determine a degree of damage to the particular property component;

determine a score indicating the degree of damage to the property component; and

determine a condition of the particular property component based on the score;

determine a level of risk of the property, wherein the level of risk is not based on the degree of damage of a property component;

assign a level of risk indicator to the property based upon the determined level of risk;

provide, for display on a user interface, an indication of the condition of each property component overlaying the corresponding portion of the one or more aerial images; and

provide the level of risk indicator for display on the user interface.

14. The non-transitory computer-readable memory of claim 13 , wherein the condition of the particular property component includes a change in condition of the particular property component over time.

15. The non-transitory computer-readable memory of claim 14 , wherein to determine the level of risk of the property, the instructions cause the one or more processors to determine the level of risk of the property based upon an amount of time in which the property component is in a particular condition.

16. The non-transitory computer-readable memory of claim 13 , wherein the instructions further cause the one or more processors to:

determine an insurance premium for the property based upon the level of risk.

17. The non-transitory computer-readable memory of claim 13 , wherein the condition of the particular property component is categorized as at least one of: excellent, good, fair, or poor.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 18, 2025
From: STATE FARM MUTUAL AUTOMOBILE INSURANCE CO.
To: NEARMAP US, INC.
Reel/Frame 070548/0732 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 3, 2023
From: HARVEY, BRIAN N.; TOFTE, NATHAN L.; SCHMIDGALL, ROGER D.; AVILES, MICHAEL JEFFREY; POTT, KYLE; GRANT, ROSEMARIE GEIER; HAEFLI, ERIC; JACOB, MICHAEL SHAWN
To: STATE FARM MUTUAL AUTOMOBILE INSURANCE COMPANY
Reel/Frame 062261/0922 →
Continuity (7)
Continuation 16831401 · Mar 26, 2020
Continuation 15966086 · Apr 30, 2018
Continuation 15718323 · Sep 28, 2017
Continuation 15165457 · May 26, 2016
Continuation 14808502 · Jul 24, 2015
Continuation 14510784 · Oct 9, 2014
Related Publication 20230081273A1 · Mar 16, 2023
References Cited (97)
US 7809587B2 · Dorai et al. · 2010 [cited by applicant]
US 8346578B1 · Hopkins, III et al. · 2013 [cited by applicant]
US 8380545B2 · Chen et al. · 2013 [cited by applicant]
US 8666786B1 · Wirz et al. · 2014 [cited by applicant]
US 8756085B1 · Plummer et al. · 2014 [cited by applicant]
US 8818572B1 · Tofte et al. · 2014 [cited by applicant]
US 8872818B2 · Freeman et al. · 2014 [cited by applicant]
US 8874454B2 · Plummer et al. · 2014 [cited by applicant]
US 9002719B2 · Tofte · 2015 [cited by applicant]
US 9129355B1 · Harvey et al. · 2015 [cited by applicant]
US 9262789B1 · Tofte · 2016 [cited by applicant]
US 9390489B1 · Harvey et al. · 2016 [cited by applicant]
US 9489696B1 · Tofte · 2016 [cited by applicant]
US 9612598B2 · Schultz et al. · 2017 [cited by applicant]
US 9633146B2 · Plummer et al. · 2017 [cited by applicant]
US 9805456B1 · Harvey et al. · 2017 [cited by applicant]
US 10102584B1 · Devereaux et al. · 2018 [cited by applicant]
US 10163162B1 · Devereaux · 2018 [cited by examiner]
US 10311302B2 · Kottenstette et al. · 2019 [cited by applicant]
US 10366288B1 · Kottenstette et al. · 2019 [cited by applicant]
US 10643072B2 · Kottenstette et al. · 2020 [cited by applicant]
US 11069050B1 · Harvey et al. · 2021 [cited by applicant]
US 11151378B2 · Kottenstette et al. · 2021 [cited by applicant]
US 11210552B2 · Kossyk et al. · 2021 [cited by applicant]
US 11222426B2 · Richter et al. · 2022 [cited by applicant]
US 11232150B2 · Vianello et al. · 2022 [cited by applicant]
US 11367265B2 · Vianello et al. · 2022 [cited by applicant]
US 20060271695A1 · Lavian · 2006 [cited by applicant]
US 20070070185A1 · Dy et al. · 2007 [cited by applicant]
US 20070260401A1 · Sydor et al. · 2007 [cited by applicant]
US 20090265193A1 · Collins et al. · 2009 [cited by applicant]
US 20120033851A1 · Chen et al. · 2012 [cited by applicant]
US 20120109690A1 · Weinrauch et al. · 2012 [cited by applicant]
US 20130004017A1 · Medasani et al. · 2013 [cited by applicant]
US 20130211790A1 · Loveland et al. · 2013 [cited by applicant]
US 20140019166A1 · Swanson et al. · 2014 [cited by applicant]
US 20140100889A1 · Tofte · 2014 [cited by applicant]
US 20140245210A1 · Battcher et al. · 2014 [cited by applicant]
US 20140259549A1 · Freeman et al. · 2014 [cited by applicant]
US 20140270492A1 · Christopulos · 2014 [cited by examiner]
US 20140278573A1 · Cook · 2014 [cited by applicant]
US 20140278587A1 · Plummer et al. · 2014 [cited by applicant]
US 20140320651A1 · Mcclatchie et al. · 2014 [cited by applicant]
US 20140324483A1 · Plummer et al. · 2014 [cited by applicant]
US 20140334689A1 · Butler et al. · 2014 [cited by applicant]
US 20150019267A1 · Prieto et al. · 2015 [cited by applicant]
US 20150073834A1 · Gurenko · 2015 [cited by examiner]
Atkins et al., “An Integrated damage, visual, and radar analysis of the 2013 Moore Oklahoma EF5 tornado,” American Meteorological Society, Jul. 2014. [cited by applicant]
Brown et al., The use of remotely sensed data and ground survey tools to assess damage and monitor early recovery following the Dec. 5, 2008 Wenchuan earthquake in China, Bull. Earthquake Eng., 10:741-64 (2012). [cited by applicant]
Brunner et al., Earthquake damage assessment of buildings using VHR optical and SAR imagery, IEEE Transactions on Geoscience and Remote Sensing, vol. 8, No. 5 (May 2010). [cited by applicant]
Chen et al., Small-format fly-over photography for highway system monitoring, Proc. of SPIE, vol. 7647, 2010. [cited by applicant]
Gong et al., Road damage detection from high-resolution RS image, IGARSS 2012. [cited by applicant]
Jengo et al., Pothole detection and road condition assessment using hyperspectral imagery, ASPRS 2005 Annual Conference. [cited by applicant]
Li et al., “Study on road damage assessment based on RS and GIS,” IGARSS 2010. [cited by applicant]
Liu et al., Object-oriented remote sensing image classification and road damage adaptive extraction, RSETE 2013. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/510,307 dated Dec. 18, 2014. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/510,307 dated Jun. 24, 2015. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/510,307 dated Jun. 26, 2017. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/510,307 dated May 20, 2016. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/510,307 dated Nov. 19, 2015. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/510,492 dated Feb. 2, 2016. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/510,492 dated Jan. 18, 2018. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/510,492 dated Jan. 8, 2015. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/510,492 dated Jul. 12, 2016. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/510,492 dated Jul. 15, 2015. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/510,536 dated Dec. 23, 2015. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/510,536 dated Jul. 6, 2015. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/510,536 dated Jun. 27, 2017. . [cited by applicant]
Office Action issued in U.S. Appl. No. 14/510,536 dated May 25, 2016. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/510,536 dated Nov. 27, 2015. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/510,784 dated Nov. 25, 2014. [cited by applicant]
Samadzadegan et al., “Earthquake destruction assessment of urban road network using satellite imagery and fuzzy interference systems,” Proc. The International Archives of the Photogrammetry, Remote Sensing and Spatial I… [cited by applicant]
U.S. Appl. No. 14/808,502, Nonfinal Office Action, dated Nov. 27, 2015. [cited by applicant]
U.S. Appl. No. 14/808,502, Notice of Allowance, dated Mar. 22, 2016. [cited by applicant]
U.S. Appl. No. 15/165,457, Notice of Allowance, dated Jul. 12, 2017. [cited by applicant]
U.S. Appl. No. 15/718,323, Nonfinal Office Action, dated Oct. 26, 2017. [cited by applicant]
U.S. Appl. No. 15/718,323, Notice of Allowance, dated Mar. 2, 2018. [cited by applicant]
U.S. Appl. No. 15/837,649, filed Dec. 11, 2017. [cited by applicant]
U.S. Appl. No. 15/966,086, Final Office Action, dated Aug. 3, 2020. [cited by applicant]
U.S. Appl. No. 15/966,086, Nonfinal Office Action, dated Mar. 30, 2020. [cited by applicant]
U.S. Appl. No. 15/966,086, Nonfinal Office Action, dated Nov. 16, 2020. [cited by applicant]
U.S. Appl. No. 15/966,086, Notice of Allowance, dated Mar. 17, 2021. [cited by applicant]
U.S. Appl. No. 16/831,401, Final Office Action, dated Jul. 20, 2022. [cited by applicant]
U.S. Appl. No. 16/831,401, Final Office Action, dated Oct. 5, 2021. [cited by applicant]
U.S. Appl. No. 16/831,401, Nonfinal Office Action, dated Dec. 1, 2022. [cited by applicant]
U.S. Appl. No. 16/831,401, Nonfinal Office Action, dated Jan. 28, 2022. [cited by applicant]
U.S. Appl. No. 16/831,401, Nonfinal Office Action, dated Mar. 19, 2021. [cited by applicant]
U.S. Appl. No. 14/510,307 entitled “Method and System for Determining the Condition of Insured Properties in a Neighborhood,” filed Oct. 9, 2014. [cited by applicant]
U.S. Appl. No. 14/510,492 entitled “Method and System for Assessing Damage to Insured Properties in a Neighborhood,” filed Oct. 9, 2014. [cited by applicant]
U.S. Appl. No. 14/510,536 entitled “Method and System for Generating Real-Time Images of Customer Homes During a Catastrophe,” filed Oct. 9, 2014. [cited by applicant]
U.S. Appl. No. 14/510,784 entitled “Method and System for Assessing Damage To Infrastructure,” filed Oct. 9, 2014. [cited by applicant]
Wang et al., An object-oriented method for road damage detection from high resolution remote sensing images, 19th International Conference on Geoinformatics, 2011. [cited by applicant]
Yamazaki, “Applications of remote sensing and GIS for damage assessment,” In: ICUS/INCEDE Report 1, Joint Workshop on Urban Safety Engineering, Bangkok, 2001. [cited by applicant]
Mackie et al., “Performance-based earthquake assessment of bridge systems including ground-foundation interaction”, Soil Dynamics and Earthquake Engineering 42 (2012) 184-196. [cited by applicant]
Thomas et al., “Automated poststorm damage classification of low-rise building roofing systems using high-resolution aerial imagery”, IEEE Transactions on Geoscience and Remote Sensing, vol. 52, No. 7, Jul. 2014. [cited by applicant]
Carunnio et al., “EDCAR: a knowledge representation framework to enhance automatic video surveillance” Expert Systems with Applications, 131, (2019), 190-207. [cited by applicant]
Hongeng et al., “Video-based event recognition: activity representation and probabilistic recognition methods”, Computer Vision and Image Understading 96, (2004), 129-162. [cited by applicant]