IP Library › Granted Patent US 12,314,635
Granted Patent B2
US 12,314,635 · App. 17/122,080 · Granted May 27, 2025

Systems and methods for rapidly developing annotated computer models of structures

Inventors: Dave Harris (Orem, UT); Ron Richardson (South Jordan, UT); Corey David Reed (Cedar Hills, UT); Jeffery Devon Lewis (Orem, UT); Jeffrey Clayton Taylor (Alpine, UT)
Assignee: Insurance Services Office, Inc.
G06F30/13G06F3/04815G06F3/0482G06F30/12G06Q30/0283G06Q40/08G06V20/20G06T11/60G06T2200/24G06T2219/004G06V20/176
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,314,635
App. No.
17/122,080
Filed
Dec 15, 2020
Granted
May 27, 2025
Kind
B2
Art Unit
2600
USPC
382/154
Abstract

Systems and methods for rapidly developing annotated computer models of structures and properties is provided. The system generates three-dimensional (3D) models of structures and property using a wide variety of digital imagery, and/or can process existing 3D models created by other systems. The system processes the 3D models to automatically identify candidate objects within the 3D models that may be suitable for annotation, such as roof faces, chimneys, windows, gutters, etc., using computer vision techniques to automatically identify such objects. Once the candidate objects have been identified, the system automatically generates user interface screens which gather relevant information related to the candidate objects, so as to rapidly obtain, associate, and store annotation information related to the candidate objects. When all relevant annotation information has been gathered and associated with model objects, the system can create a list of materials that can be used for future purposes, such as repair and/or reconstruction of real-world structures and property. The system also allows for modeling of water damage of a structure, as well as generating lists of tasks for mitigating the water damage and associated costs.

Claims (26)

1. A method for rapidly developing an annotated computer model of water damage relating to a structure, comprising the steps of:

generating a computerized floorplan model of a structure in a user interface of a computer system;

graphically defining at least one area within the floorplan model corresponding to a region where water has damaged the structure;

graphically depicting the at least one area in the floorplan model in the user interface;

automatically generating and displaying in the user interface a plurality of questions to be answered by a user of the system relating to the at least one area within the floorplan model;

allowing the user to answer the plurality of questions relating to the at least one area within the floorplan model; and

generating a list of actions to be taken to mitigate water damage in the at least one area of the structure.

2. The method of claim 1 , further comprising allowing the user to define a plurality of areas within the floorplan model corresponding to regions where water has damaged the structure.

3. The method of claim 2 , further comprising allowing the user to generate a zone encompassing the plurality of areas.

4. The method of claim 2 , further comprising allowing the user to define one or more boundaries between the plurality of areas using a boundary tool.

5. The method of claim 1 , wherein plurality of questions are presented to the user in a guided script.

6. The method of claim 1 , further comprising displaying a user interface tool to the user allowing the user to perform a calculation relating to a piece of equipment that can be utilized to mitigate water damage.

7. A system for rapidly developing an annotated computer model of water damage relating to a structure, comprising:

a computer system having a memory and a display; and

a processor of the computer system, the processor programmed to perform steps of:

generating a computerized floorplan model of a structure in a user interface of a computer system;

graphically defining at least one area within the floorplan model corresponding to a region where water has damaged the structure;

graphically depicting the at least one area in the floorplan model in the user interface;

automatically generating and displaying in the user interface a plurality of questions to be answered by a user of the system relating to the at least one area within the floorplan model;

allowing the user to answer the plurality of questions relating to the at least one area within the floorplan model; and

generating a list of actions to be taken to mitigate water damage in the at least one area of the structure.

8. The system of claim 7 , the processor allowing the user to define a plurality of areas within the floorplan model corresponding to regions where water has damaged the structure.

9. The system of claim 8 , the processor allowing the user to generate a zone encompassing the plurality of areas.

10. The system of claim 8 , the processor allowing the user to define one or more boundaries between the plurality of areas using a boundary tool.

11. The system of claim 7 , wherein plurality of questions are presented to the user in a guided script.

12. The system of claim 7 , the processor displaying a user interface tool to the user allowing the user to perform a calculation relating to a piece of equipment that can be utilized to mitigate water damage.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 15, 2021
From: HARRIS, DAVE; RICHARDSON, RON; REED, COREY DAVID; LEWIS, JEFFERY DEVON; TAYLOR, JEFFREY CLAYTON
To: INSURANCE SERVICES OFFICE, INC.
Reel/Frame 055730/0649 →
Continuity (3)
Continuation In Part 16189512 · Nov 13, 2018
Provisional Application 62585078 · Nov 13, 2017
Related Publication 20210103687A1 · Apr 8, 2021
References Cited (175)
US 5701403A · Watanabe et al. · 1997 [cited by applicant]
US 6446030B1 · Hoffman et al. · 2002 [cited by applicant]
US 6448964B1 · Saacs et al. · 2002 [cited by applicant]
US 8533063B2 · Erickson · 2013 [cited by applicant]
US 8843304B1 · Dupont et al. · 2014 [cited by applicant]
US 8868375B1 · Christian · 2014 [cited by applicant]
US 8983806B2 · Labrie et al. · 2015 [cited by applicant]
US 9158869B2 · Labrie et al. · 2015 [cited by applicant]
US 9430871B2 · Neophytou et al. · 2016 [cited by applicant]
US 9501700B2 · Loveland et al. · 2016 [cited by applicant]
US 9679227B2 · Taylor et al. · 2017 [cited by applicant]
US 10127670B2 · Lewis et al. · 2018 [cited by applicant]
US 10181079B2 · Labrie et al. · 2019 [cited by applicant]
US 10289760B1 · Oakes, III et al. · 2019 [cited by applicant]
US 10387582B2 · Lewis et al. · 2019 [cited by applicant]
US 10445438B1 · Motonaga et al. · 2019 [cited by applicant]
US 10521865B1 · Spader et al. · 2019 [cited by applicant]
US 10529028B1 · Davis et al. · 2020 [cited by applicant]
US 11314905B2 · Childs et al. · 2022 [cited by applicant]
US 11688135B2 · Lewis et al. · 2023 [cited by applicant]
US 11688186B2 · Harris et al. · 2023 [cited by applicant]
US 11734468B2 · Lewis et al. · 2023 [cited by applicant]
US 20020116254A1 · Stein et al. · 2002 [cited by applicant]
US 20030009315A1 · Thomas et al. · 2003 [cited by applicant]
US 20070080961A1 · Inzinga et al. · 2007 [cited by applicant]
US 20070276626A1 · Bruffey · 2007 [cited by applicant]
US 20090179895A1 · Zhu et al. · 2009 [cited by applicant]
US 20100110074A1 · Pershing · 2010 [cited by applicant]
US 20100114537A1 · Pershing · 2010 [cited by applicant]
US 20100296693A1 · Thornberry et al. · 2010 [cited by applicant]
US 20110056286A1 · Jansen · 2011 [cited by applicant]
US 20110157213A1 · Takeyama et al. · 2011 [cited by applicant]
US 20110191738A1 · Walker et al. · 2011 [cited by applicant]
US 20120026322A1 · Malka et al. · 2012 [cited by applicant]
US 20120179431A1 · Labrie et al. · 2012 [cited by applicant]
US 20120253725A1 · Malka et al. · 2012 [cited by applicant]
US 20120253751A1 · Malka et al. · 2012 [cited by applicant]
US 20130201167A1 · Oh et al. · 2013 [cited by applicant]
US 20130206177A1 · Burlutskiy · 2013 [cited by applicant]
US 20130226451A1 · O'Neill et al. · 2013 [cited by applicant]
US 20130262029A1 · Pershing · 2013 [cited by applicant]
US 20130267260A1 · Chao et al. · 2013 [cited by applicant]
US 20130314688A1 · Likholyot · 2013 [cited by applicant]
US 20140043436A1 · Bell et al. · 2014 [cited by applicant]
US 20140195275A1 · Pershing et al. · 2014 [cited by applicant]
US 20140279593A1 · Pershing · 2014 [cited by examiner]
US 20140301633A1 · Furukawa et al. · 2014 [cited by applicant]
US 20140320661A1 · Sankar et al. · 2014 [cited by applicant]
US 20150029182A1 · Sun et al. · 2015 [cited by applicant]
US 20150073864A1 · Labrie et al. · 2015 [cited by applicant]
US 20150093047A1 · Battcher et al. · 2015 [cited by applicant]
US 20150116509A1 · Birkler et al. · 2015 [cited by applicant]
US 20150153172A1 · Starns et al. · 2015 [cited by applicant]
US 20150193971A1 · Dryanovski et al. · 2015 [cited by applicant]
US 20150213558A1 · Nelson · 2015 [cited by applicant]
US 20150227645A1 · Childs et al. · 2015 [cited by applicant]
US 20150269438A1 · Samarasekera et al. · 2015 [cited by applicant]
US 20150302529A1 · Jagannathan · 2015 [cited by applicant]
US 20160098802A1 · Bruffey et al. · 2016 [cited by applicant]
US 20160110480A1 · Randolph · 2016 [cited by applicant]
US 20160246767A1 · Makadia et al. · 2016 [cited by applicant]
US 20160282107A1 · Roland et al. · 2016 [cited by applicant]
US 20170124713A1 · Jurgenson et al. · 2017 [cited by applicant]
US 20170132711A1 · Bruffey et al. · 2017 [cited by applicant]
US 20170132835A1 · Halliday et al. · 2017 [cited by applicant]
US 20170169459A1 · Bruffey et al. · 2017 [cited by applicant]
US 20170193297A1 · Michini et al. · 2017 [cited by applicant]
US 20170206648A1 · Marra et al. · 2017 [cited by applicant]
US 20170221152A1 · Nelson et al. · 2017 [cited by applicant]
US 20170316115A1 · Lewis et al. · 2017 [cited by applicant]
US 20170330207A1 · Labrie et al. · 2017 [cited by applicant]
US 20170345069A1 · Labrie et al. · 2017 [cited by applicant]
US 20180053329A1 · Roberts et al. · 2018 [cited by applicant]
US 20180067593A1 · Tiwari et al. · 2018 [cited by applicant]
US 20180089833A1 · Lewis et al. · 2018 [cited by applicant]
US 20180286098A1 · Lorenzo · 2018 [cited by applicant]
US 20180330528A1 · Loveland et al. · 2018 [cited by applicant]
US 20180357819A1 · Oprea · 2018 [cited by applicant]
US 20180373931A1 · Li · 2018 [cited by applicant]
US 20190114717A1 · Labrie et al. · 2019 [cited by applicant]
US 20190147247A1 · Harris et al. · 2019 [cited by applicant]
US 20190221040A1 · Shantharam et al. · 2019 [cited by applicant]
US 20190340692A1 · Labrie et al. · 2019 [cited by applicant]
US 20190347859A1 · Jovanovic et al. · 2019 [cited by applicant]
US 20190377837A1 · Lewis et al. · 2019 [cited by applicant]
US 20200100066A1 · Lewis et al. · 2020 [cited by applicant]
US 20200143481A1 · Brown et al. · 2020 [cited by applicant]
US 20210076162A1 · Wang et al. · 2021 [cited by applicant]
US 20210350038A1 · Jenson et al. · 2021 [cited by applicant]
US 20220309204A1 · Childs et al. · 2022 [cited by applicant]
US 20220309748A1 · Lewis et al. · 2022 [cited by applicant]
US 20220414980A1 · Lewis et al. · 2022 [cited by applicant]
US 20230343030A1 · Lewis et al. · 2023 [cited by applicant]
US 20230409769A1 · Lewis et al. · 2023 [cited by applicant]
WO 2014151122A1 · 2014 [cited by applicant]
WO 2016154306A1 · 2016 [cited by applicant]
WO 2017100658A1 · 2017 [cited by applicant]
Sampaio et al. “Maintenance of Building Components Supported in Interactive Environments.” 10th Iberian Conference on Information Systems and Technologies, Jun. 17, 2015, DOI: 10.1109/CISTI.2015.7170514, 6 pages (Year: … [cited by examiner]
Office Action dated Sep. 2, 2022, issued in connection with U.S. Appl. No. 16/580,741 (13 pages). [cited by applicant]
Notice of Allowance mailed Sep. 6, 2022, issued in connection with U.S. Appl. No. 16/189,512 (7 pages). [cited by applicant]
Office Action mailed Oct. 13, 2022, issued in connection with U.S. Appl. No. 16/545,607 (53 pages). [cited by applicant]
International Search Report of the International Searching Authority mailed on Nov. 18, 2022, issued in connection with International Application No. PCT/US22/030691 (6 pages). [cited by applicant]
Written Opinion of the International Searching Authority mailed on Nov. 18, 2022, issued in connection with International Application No. PCT/US22/030691 (11 pages). [cited by applicant]
Notice of Allowance mailed Dec. 9, 2022, issued in connection with U.S. Appl. No. 17/705,130 (10 pages). [cited by applicant]
Examination Report No. 1 mailed Dec. 15, 2022, issued by the Australian Patent Office in connection with Australian Patent Application No. 2021282413 (3 pages). [cited by applicant]
International Search Report of the International Searching Authority mailed on Mar. 27, 2017, issued in connection with International Application No. PCT/US2016/65947 (3 pages). [cited by applicant]
Written Opinion of the International Searching Authority mailed on Mar. 27, 2017, issued in connection with International Application No. PCT/US2016/65947 (7 pages). [cited by applicant]
Office Action mailed Sep. 26, 2018, issued in connection with U.S. Appl. No. 15/374,695 (33 pages). [cited by applicant]
Notice of Allowance mailed May 13, 2019, issued in connection with U.S. Appl. No. 15/374,695 (7 pages). [cited by applicant]
Extended European Search Report mailed Jun. 11, 2019, issued in connection with European Patent Application No. 16873975.3 (8 pages). [cited by applicant]
Communication Pursuant to Article 94(3) EPC issued by the European Patent Office mailed on Apr. 22, 2020, issued In connection with European Patent Application No. 16873975.3 (6 pages). [cited by applicant]
International Search Report of the International Searching Authority mailed on Dec. 12, 2019, issued in connection with International Application No. PCT/US2019/52670 (3 pages). [cited by applicant]
Written Opinion of the International Searching Authority mailed on Dec. 12, 2019, issued in connection with International Application No. PCT/US2019/52670 (5 pages). [cited by applicant]
Office Action mailed Feb. 5, 2020, issued in connection with U.S. Appl. No. 16/580,741 (15 pages). [cited by applicant]
International Search Report of the International Searching Authority mailed on May 14, 2015, issued in connection with International Application No. PCT/US15/015491(3 pages). [cited by applicant]
Written Opinion of the International Searching Authority mailed on May 14, 2015, issued in connection with International Application No. PCT/US15/015491 (9 pages). [cited by applicant]
Fung, et al., “A Mobile Assisted Localization Scheme for Augmented Reality,” Department of Computer Science and Engineering, The Chinese University of Hong Kong, 2012 (76 pages). [cited by applicant]
Sankar, et al., “Capturing Indoor Scenes With Smartphones,” UIST' 12, Oct. 7-10, 2012, Cambridge, Massachusetts (9 pages). [cited by applicant]
Office Action mailed Aug. 8, 2017, issued in connection with U.S. Appl. No. 14/620,004 (26 pages). [cited by applicant]
Office Action mailed Aug. 28, 2018, issued in connection with U.S. Appl. No. 14/620,004 (33 pages). [cited by applicant]
Farin, et al., “Floor-Plan Reconstruction from Panoramic Images,” Sep. 23-28, 2007, MM '07, ACM (4 pages). [cited by applicant]
Office Action mailed Mar. 29, 2019, issued in connection with U.S. Appl. No. 14/620,004 (22 pages). [cited by applicant]
Office Action mailed Dec. 10, 2019, issued in connection with U.S. Appl. No. 14/620,004 (27 pages). [cited by applicant]
Zhang, et al., “Wallk&Sketch: Create Floor Plans with an RGB-D Camera,” Sep. 5-8, 2012, UbiComp '12, ACM (10 pages). [cited by applicant]
Office Action mailed Jul. 8, 2020, issued in connection with U.S. Appl. No. 14/620,004 (27 pages). [cited by applicant]
Office Action mailed Sep. 22, 2020, issued in connection with U.S. Appl. No. 16/580,741 (14 pages). [cited by applicant]
Office Action mailed Feb. 2, 2021, issued in connection with U.S. Appl. No. 14/620,004 (28 pages). [cited by applicant]
Communication Pursuant to Article 94(3) EPC issued by the European Patent Office mailed on Feb. 18, 2021, Issued in connection with European Patent Application No. 16873975.3 (5 pages). [cited by applicant]
Examination Report No. 1 mailed Mar. 30, 2021, issued by the Australian Patent Office in connection with Australian Patent Application No. 2016366537 (6 pages). [cited by applicant]
Office Action mailed Apr. 21, 2021, issued in connection with U.S. Appl. No. 16/580,741 (15 pages). [cited by applicant]
Notice of Allowance mailed Aug. 19, 2021, issued in connection with U.S. Appl. No. 14/620,004 (11 pages). [cited by applicant]
Examiner-Initiated Interview Summary dated Aug. 10, 2021, issued in connection with U.S. Appl. No. 14/620,004 (1 page). [cited by applicant]
Communication Pursuant to Article 94(3) EPC mailed Jan. 31, 2023, issued in connection with European Patent Application No. 16873975.3 (8 pages). [cited by applicant]
Notice of Allowance mailed Feb. 14, 2023, issued in connection with U.S. Appl. No. 17/705,130 (5 pages). [cited by applicant]
Office Action dated Jan. 12, 2024, issued in connection with U.S. Appl. No. 18/214,842 (4 pages). [cited by applicant]
International Search Report of the International Searching Authority mailed on Dec. 15, 2023, issued in connection with International Application No. PCT/US2023/031123 (6 pages). [cited by applicant]
Written Opinion of the International Searching Authority mailed on Dec. 15, 2023, issued in connection with International Application No. PCT/US2023/031123 (7 pages). [cited by applicant]
Office Action mailed Dec. 27, 2021, issued in connection with U.S. Appl. No. 16/580,741 (13 pages). [cited by applicant]
Notice of Allowance mailed Dec. 16, 2021, issued in connection with U.S. Appl. No. 14/620,004 (12 pages). [cited by applicant]
Invitation to Pay Additional Fees issued by the International Searching Authority mailed on Feb. 2, 2022, issued in connection with International Application No. PCT/US21/63469 (2 pages). [cited by applicant]
Extended European Search Report dated Feb. 18, 2022, issued in connection with European Patent Application No. 19866788.3 (9 pages). [cited by applicant]
Communication Pursuant to Article 94(3) EPC issued by the European Patent Office mailed on Apr. 28, 2023, issued In connection with European Patent Application No. 19866788.3 (5 pages). [cited by applicant]
Office Action mailed Jun. 30, 2023, issued in connection with U.S. Appl. No. 17/729,613 (49 pages). [cited by applicant]
Notice of Allowance mailed Mar. 22, 2023, issued in connection with U.S. Appl. No. 16/545,607 (9 pages). [cited by applicant]
International Search Report of the International Searching Authority mailed on Apr. 8, 2022, issued in connection with International Application No. PCT/US21/63469 (5 pages). [cited by applicant]
Written Opinion of the International Searching Authority mailed on Apr. 8, 2022, issued in connection with International Application No. PCT/US21/63469 (6 pages). [cited by applicant]
Office Action mailed Mar. 25, 2022, issued in connection with U.S. Appl. No. 16/545,607 (56 pages). [cited by applicant]
Notice of Allowance mailed Apr. 8, 2022, issued in connection with U.S. Appl. No. 16/189,512 (8 pages). [cited by applicant]
Notice of Allowance mailed Jun. 21, 2022, issued in connection with U.S. Appl. No. 16/189,512 (7 pages). [cited by applicant]
Dino, et al., “Image-Based Construction of Building Energy Models Using Computer Vision,” Automation in Construction (2020) (15 pages). [cited by applicant]
Fathi, et al., “Automated As-Built 3D Reconstruction of Civil Infrastructure Using Computer Vision: Achievements, Opportunities, and Challenges,” Advanced Engineering Informatics (2015) (13 pages). [cited by applicant]
International Search Report of the International Searching Authority mailed on Jul. 25, 2022, issued in connection with International Application No. PCT/US22/22024 (3 pages). [cited by applicant]
Written Opinion of the International Searching Authority mailed on Jul. 25, 2022, issued in connection with International Application No. PCT/US22/22024 (5 pages). [cited by applicant]
International Search Report of the International Searching Authority mailed on Feb. 11, 2019, issued in connection with International Application No. PCT/US18/60762 (3 pages). [cited by applicant]
Written Opinion of the International Searching Authority mailed on Feb. 11, 2019, issued in connection with International Application No. PCT/US18/60762 (7 pages). [cited by applicant]
Office Action mailed Apr. 16, 2020, issued in connection with U.S. Appl. No. 16/189,512 (10 pages). [cited by applicant]
U.S. Appl. No. 62/512,989, filed May 31, 2017 entiitled, “Systems and Methods for Rapidly Developing Annotated Computer Models of Structures” (47 pages). [cited by applicant]
Office Action mailed Dec. 14, 2020, issued in connection with U.S. Appl. No. 16/189,512 (10 pages). [cited by applicant]
Extended European Search Report dated Jul. 1, 2021, issued by the European Patent Office in connection with European Application No. 18876121.7 (8 pages). [cited by applicant]
Office Action mailed Jul. 20, 2021, issued in connection with U.S. Appl. No. 16/189,512 (11 pages). [cited by applicant]
Examination Report No. 1 mailed Apr. 12, 2023, issued by the Australian Patent Office in connection with Australian Patent Application No. 2018364813 (3 pages). [cited by applicant]
Office Action mailed Dec. 7, 2023, issued by the Canadian Patent Office in connection with Canadian Patent Application No. 3,082,516 (6 pages). [cited by applicant]
Office Action dated Jan. 30, 2024, issued in connection with U.S. Appl. No. 17/895,500 (16 pages). [cited by applicant]
Office Action dated Mar. 14, 2024, issued in connection with U.S. Appl. No. 18/236,728 (58 pages). [cited by applicant]
Notice of Allowance dated Mar. 18, 2024, issued in connection with U.S. Appl. No. 17/729,613 (10 pages). [cited by applicant]
Communication Pursuant to Article 94(3) EPC mailed Jan. 23, 2024, issued by the European Patent Office in connection with European Patent Application No. 19866788.3 (5 pages). [cited by applicant]
Office Action dated Jun. 4, 2024, issued by the Canadian Intellectual Property Office in connection with Canadian Application No. 3,082,516 (4 pages). [cited by applicant]
Notice of Allowance dated Jul. 3, 2024, issued in connection with U.S. Appl. No. 17/895,500 (6 pages). [cited by applicant]
Notice of Allowance dated Jul. 12, 2024, issued in connection with U.S. Appl. No. 17/729,613 (9 pages). [cited by applicant]
Extended European Search Report dated Oct. 14, 2024, issued by the European Patent Office in connection with European Application No. 21907688.2 (11 pages). [cited by applicant]
Communication Pursuant to Article 94(3) EPC issued by the European Patent Office mailed on Dec. 19, 2024, issued in connection with European Patent Application No. 18876121.7 (4 pages). [cited by applicant]
Office Action dated Jan. 15, 2025, issued in connection with U.S. Appl. No. 17/329,098 (11 pages). [cited by applicant]
Examiner's Report dated Feb. 15, 2025, issued by the Canadian Intellectual Property Office in connection with Canadian Application No. 3,082,516 (6 pages). [cited by applicant]
Annotely—Image Annotation Made Simple. Annotate Any Image Anywhere, <https://annotely.com/releases> (2010) (18 pages). [cited by applicant]
Partial Supplementary European Search Report dated Mar. 26, 2025, issued by the European Patent Office in connection with European Application No. 22811964.0 (13 pages). [cited by applicant]