IP Library › Granted Patent US 12,198,364
Granted Patent B2
US 12,198,364 · App. 17/499,258 · Granted Jan 14, 2025

Computer vision systems and methods for detecting and modeling features of structures in images

Inventors: Jeffery Devon Lewis (Orem, UT); Bryce Zachary Porter (Lehi, UT); Ryan Mark Justus (Lehi, UT)
Assignee: Xactware Solutions, Inc.
G06T7/564G06F16/532G06T7/12G06T7/536G06T2207/10032G06T2207/20044
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,198,364
App. No.
17/499,258
Filed
Oct 12, 2021
Granted
Jan 14, 2025
Kind
B2
Examiner
ROZ, MARK
Art Unit
2675
USPC
382/154
Abstract

A computer vision system and method for detecting and modeling features of a building in a plurality of images is provided. The system includes at least one computer system in communication with a database of aerial imagery, and computer vision system code executed by the at last one computer system which automatically detects contours and infers interior roof features of the building. The system first processes the plurality of images to identify a plurality of two-dimensional (2D) line segments in each image. Then, the system processes the plurality of 2D line segments to generate a plurality of three-dimensional (3D) line segments. The plurality of 2D line segments are then processed to detect a contour of the structure, and the contour of the structure is utilized by the system to infer interior roof lines from the structure. A model of the roof of the structure is finally generated using the detected contour and interior roof lines. The system can execute a straight-skeleton algorithm to infer the interior roof lines from the detected contour. Additionally, the system can detect roof rakes from the images.

Claims (54)

1. A computer vision system for detecting and modeling features of a building in a plurality of images, comprising:

at least one computer system in communication with a database of aerial imagery; and

computer vision system code executed by said at least one computer system, said computer vision system code causing said computer system to:

receive a plurality of images, each of said images including a view of a structure;

process the plurality of images to detect a contour of the structure;

process the contour of the structure to infer interior roof lines from the detected contour; and

generate a model of the roof of the structure using the detected contour and the interior roof lines,

wherein the step of processing the plurality of images further comprises processing a plurality of two-dimensional (2D) line segments identified in the plurality of images to generate a plurality of three-dimensional (3D) line segments, and

wherein the computer vision system code causes said computer system to covert pixel coordinates of the 2D line segments to world space coordinates.

2. The system of claim 1 , wherein the step of processing the plurality of images further comprises processing the plurality of 3D line segments to detect the contour of the structure.

3. The system of claim 1 , wherein the plurality of images comprise a plurality of aerial images.

4. The system of claim 1 , wherein the computer vision system code causes said computer system to post-process the 2D lines to remove undesired 2D lines which do not correspond to a structure of interest.

5. The system of claim 4 , wherein the system further comprises an edge detection network for post-processing the 2D lines.

6. The system of claim 5 , wherein the system further comprises a building segmentation network for post-processing the 2D lines.

7. The system of claim 6 , wherein the holistic edge detection network generates a building edge mask for post-processing the 2D lines.

8. The system of claim 6 , wherein the building segmentation network generates a building segmentation mask for post-processing the 2D lines.

9. The system of claim 1 , wherein the computer vision system code causes said computer system to process 3D line segments using cluster-based detection.

10. The system of claim 9 , wherein the computer vision system code causes said computer system to refine the 3D line segments by projecting the 3D line segments onto a 2D mask.

11. The system of claim 1 , wherein the computer vision system code causes said computer system to detect roof rake features of the structure.

12. The system of claim 1 , wherein step of processing the plurality of images to detect the contour of the structure uses a contour inference algorithm.

13. A computer vision method for detecting and modeling features of a building in a plurality of images, comprising the steps of:

receiving at a computer system a plurality of images, each of said images including a view of a structure;

processing the plurality of images to detect a contour of the structure;

processing the contour of the structure to infer interior roof lines from the detected contour;

generating a model of the roof of the structure using the detected contour and the interior roof lines,

wherein the step of processing the plurality of images further comprises processing a plurality of two-dimensional (2D) line segments identified in the plurality of images to generate a plurality of three-dimensional (3D) line segments; and

converting pixel coordinates of the 2D line segments to world space coordinates.

14. The method of claim 13 , wherein the step of processing the plurality of images further comprises processing the plurality of 3D line segments to detect the contour of the structure.

15. The method of claim 13 , further comprising post-processing the 2D lines to remove undesired 2D lines which do not correspond to a structure of interest.

16. The method of claim 15 , further comprising applying an edge detection network for post-processing the 2D lines.

17. The method of claim 16 , further comprising applying a building segmentation network for post-processing the 2D lines.

18. The method of claim 17 , further comprising generating a building edge mask for post-processing the 2D lines.

19. The method of claim 17 , further comprising generating a building segmentation mask for post-processing the 2D lines.

20. The method of claim 13 , further comprising processing 3D line segments using cluster-based detection.

21. The method of claim 20 , further comprising refining the 3D line segments by projecting the 3D line segments onto a 2D mask.

22. The method of claim 13 , further comprising detecting roof rake features of the structure.

23. The method of claim 13 , wherein processing the plurality of images to detect the contour of the structure uses a contour inference algorithm.

24. A non-transitory, computer-readable medium having computer-readable instructions stored thereon which, when executed by a computer system, causes the computer system to execute the steps of:

receiving at the computer system a plurality of images, each of said images including a view of a structure;

processing the plurality of images to detect a contour of the structure;

processing the contour of the structure to infer interior roof lines from the detected contour;

generating a model of the roof of the structure using the detected contour and the interior roof lines,

wherein the step of processing the plurality of images further comprises processing a plurality of two-dimensional (2D) line segments to generate a plurality of three-dimensional (3D) line segments; and

converting pixel coordinates of the 2D line segments to world space coordinates.

25. The computer-readable medium of claim 4 , wherein the step of processing the plurality of images further comprises processing the plurality of 3D line segments to detect the contour of the structure.

26. The computer-readable medium of claim 24 , further comprising post-processing the 2D lines to remove undesired 2D lines which do not correspond to a structure of interest.

27. The computer-readable medium of claim 26 , further comprising applying an edge detection network for post-processing the 2D lines.

28. The computer-readable medium of claim 27 , further comprising applying a building segmentation network for post-processing the 2D lines.

29. The computer-readable medium of claim 28 , further comprising generating a building edge mask for post-processing the 2D lines.

30. The computer-readable medium of claim 28 , further comprising generating a building segmentation mask for post-processing the 2D lines.

31. The computer-readable medium of claim 24 , further comprising processing 3D line segments using cluster-based detection.

32. The computer-readable medium of claim 31 , further comprising refining the 3D line segments by projecting the 3D line segments onto a 2D mask.

33. The computer-readable medium of claim 24 , further comprising detecting roof rake features of the structure.

34. The computer-readable medium of claim 24 , wherein processing the plurality of images to detect the contour of the structure uses a contour inference algorithm.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 12, 2021
From: LEWIS, JEFFERY DEVON; PORTER, BRYCE ZACHARY; JUSTUS, RYAN MARK
To: XACTWARE SOLUTIONS, INC.
Reel/Frame 057766/0104 →
Continuity (3)
Continuation 16189339 · Nov 13, 2018
Continuation 15277359 · Sep 27, 2016
Related Publication 20220028163A1 · Jan 27, 2022
References Cited (54)
US 5671343A · Kondo · 1997 [cited by examiner]
US 8078436B2 · Pershing et al. · 2011 [cited by applicant]
US 8131514B2 · Royan et al. · 2012 [cited by applicant]
US 8170840B2 · Pershing · 2012 [cited by applicant]
US 8209152B2 · Pershing · 2012 [cited by applicant]
US 8417061B2 · Kennedy et al. · 2013 [cited by applicant]
US 8422825B1 · Neophytou et al. · 2013 [cited by applicant]
US 8731234B1 · Ciarcia et al. · 2014 [cited by applicant]
US 8818770B2 · Pershing · 2014 [cited by applicant]
US 8825454B2 · Pershing · 2014 [cited by applicant]
US 9129376B2 · Pershing · 2015 [cited by applicant]
US 9135737B2 · Pershing · 2015 [cited by applicant]
US 9501700B2 · Loveland et al. · 2016 [cited by applicant]
US 9679227B2 · Taylor et al. · 2017 [cited by applicant]
US 10127670B2 · Lewis et al. · 2018 [cited by applicant]
US 11145073B2 · Lewis et al. · 2021 [cited by applicant]
US 20110033110A1 · Shimamura · 2011 [cited by examiner]
US 20130211790A1 · Loveland et al. · 2013 [cited by applicant]
US 20180089833A1 · Lewis et al. · 2018 [cited by applicant]
US 20190102897A1 · Lewis et al. · 2019 [cited by applicant]
EP 1010966B1 · 2002 [cited by applicant]
Office Action dated Oct. 20, 2023, issued by the Canadian Patent Office in connection with Canadian Patent Application No. 3,037,360 (5 pages). [cited by applicant]
Extended European Search Report dated Dec. 18, 2023, issued by the European Patent Office in connection with European Patent Application No. 23210741.7 (12 pages). [cited by applicant]
International Search Report of the International Searching Authority mailed on Jun. 11, 2018, issued in connection with International Application No. PCT/US17/53631 (4 pages). [cited by applicant]
Written Opinion of the International Searching Authority mailed on Jun. 11, 2018, issued in connection with International Application No. PCT/US17/53631 (4 pages). [cited by applicant]
Aichholzer et al., “A Novel Type of Skeleton for Polygons”, Journal of Universal Computer Science, vol. 1, No. 12, 1995, pp. 752-761 (10 pages). [cited by applicant]
Applicad, “Product Bulletin—Nov. 2002: Key Features of Our Roofing Software,” Nov. 2002 (46 pages). [cited by applicant]
Avrahami et al., “Extraction of 3D Spatial Polygons Based on the Overlapping Criterion for Roof Extraction from Aerial Images”, CMRT05, International Archives of Photogrammetry, Remote Sensing and Spatial Information Sc… [cited by applicant]
Biedl et al., “Weighted Straight Skeletons in the Plane”, Computational Geometry, vol. 48, No. 2, 2015, pp. 120-133 (28 pages) (retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4235777/). [cited by applicant]
Felkel et al., “Straight Skeleton Implementation”, Proceedings of Spring Conference on Computer Graphics, Budmerice, Slovakia, 1998, pp. 210-218 (10 pages). [cited by applicant]
Hsieh, “Design and Evaluation of a Semi-Automated Site Modeling System”, Carnegie Mellon University, School of Computer Science, Nov. 1995 (83 pages). [cited by applicant]
Kelly, “Unwritten Procedural Modeling with the Straight Skeleton”, Doctoral dissertation, University of Glasgow, 2014 (258 pages). [cited by applicant]
Labe et al., “Robust Techniques for Estimating Parameters of 3D Building Primitives”, International Archives of Photogrammetry and Remote Sensing, vol. 32, part 2, Proceedings of the Commission II Symposium on Data Inte… [cited by applicant]
Long et al., “Fully Convolutional Networks for Semantic Segmentation”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431-3440 (10 pages). [cited by applicant]
McKeown et al., “Automatic Cartographic Feature Extraction Using Photogrammetric Principles”, Digital Photogrammetry: an Addendum to the Manual of Photogrammetry, Chapter 9: Feature Extraction and Object Recognition, Am… [cited by applicant]
Noronha et al., “Detection and Modeling of Buildings from Multiple Aerial Images,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, No. 5, May 2001, pp. 501-518 (32 pages). [cited by applicant]
Von Gioi et al., “LSD: a Line Segment Detector”, Image Processing on Line, vol. 2, 2012, pp. 35-55 (21 pages). [cited by applicant]
Xie et al., “Holistically-Nested Edge Detection”, Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1395-1403 (9 pages). [cited by applicant]
Notice of Allowance mailed Jul. 23, 2018, issued in connection with U.S. Appl. No. 15/277,359 (9 pages). [cited by applicant]
European Search Report dated Aug. 14, 2019, issued by the European Patent Office in connection with European Patent Application No. 17886999.6 (11 pages). [cited by applicant]
Shao, et al., “Automatic Building Outline Reconstruction Using 2D Building Data and Stereo Images,” Asian Conference on Remote Sensing, Nov. 25, 2002 (7 pages). [cited by applicant]
Henricsson, “The Role of Color Attributes and Similarity Grouping in 3-D Building Reconstruction,” Computer Vision and Image Understanding, Academic Press, vol. 72, No. 2, Nov. 1998 (22 pages). [cited by applicant]
Rafael Grompone von Gioi, et al., “LSD: A Line Segment Detector,” Image Processing on Line, Mar. 24, 2012 (10 pages). [cited by applicant]
Laycock, et al., “Automatically Generating Roof Models from Building Footprints,” WSCG Posters Proceedings, Feb. 3, 2003 (4 pages). [cited by applicant]
Office Action dated Apr. 1, 2020, issued in connection with U.S. Appl. No. 16/189,339 (10 pages). [cited by applicant]
Applicant-Initiated Interview Summary dated May 11, 2020, issued in connection with U.S. Appl. No. 16/189,339 (3 pages). [cited by applicant]
Office Action dated Nov. 6, 2020, issued in connection with U.S. Appl. No. 16/189,339 (10 pages). [cited by applicant]
Applicant-Intiiated Interview Summary dated Feb. 23, 2021, issued in connection with U.S. Appl. No. 16/189,339 (2 pages). [cited by applicant]
Communication Pursuant to Article 94(3) EPC dated May 6, 2021, issued by the European Patent Office in connection with European Patent Application No. 17886999.6 (6 pages). [cited by applicant]
Notice of Allowance dated Jun. 23, 2021, issued in connection with U.S. Appl. No. 16/189,339 (9 pages). [cited by applicant]
Laycock, et al., “Automatically Generating Roof Models from Building Footprints,” published in WSCG 2003 (4 pages). [cited by applicant]
Examination Report No. 1 dated Sep. 30, 2021, issued by the Australian Patent Office in connection with Australian Patent Application No. 2017387638 (5 pages). [cited by applicant]
Examination Report No. 1 dated Sep. 8, 2023, issued by the Australian Patent Office in connection with Australian Patent Application No. 2022275479 (5 pages). [cited by applicant]
Office Action dated Nov. 13, 2024, issued by the Canadian Patent Office in connection with Canadian Patent Application No. 3,037,360 (5 pages). [cited by applicant]