IP Library Granted Patent US 12,190,446
Granted Patent B2
US 12,190,446 · App. 17/993,377 · Granted Jan 7, 2025

Image capture for a multi-dimensional building model

Inventors: Manish Upendran (San Francisco, CA); William Castillo (San Francisco, CA); Ajay Mishra (San Francisco, CA); Adam J. Altman (San Francisco, CA)
Assignee: Hover Inc.
G06T17/05G06T7/55G06T7/60G06T17/00G06T19/20H04N23/63H04N23/64G06T2210/04
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,190,446
App. No.
17/993,377
Granted
Jan 7, 2025
Kind
B2
Abstract

A process for receiving, from a computing device, a series of captured building images. The process continues by processing, in real-time, each building image in the series of captured building images to determine if each building image meets a minimum criterion, wherein the minimum criteria includes applicability to be used in constructing a specific digital multi-dimensional building model. The process continues by aggregating each image meeting the minimum criteria, determining when a base set of building images has been aggregated, wherein the base set of building images includes a threshold number images to model at least a partial multi-dimensional building model representing the series of captured building images, determining one or more facades present in the partial multi-dimensional building model, determining preliminary dimensions for one or more architectural features of the one or more facades and returning, incrementally (in real-time), the preliminary dimensions to the computing device.

Claims (50)

1. A method of minimizing image capture for multi-dimensional reconstruction, the method comprising:

receiving a request for a multi-dimensional building model based on a minimum criteria, wherein the minimum criteria establishes building images necessary to produce the multi-dimensional building model;

processing a first building image of a building object and a second building image of the building object, wherein processing the first building image and the second building image comprises determining, by one or more processors, if a first minimum criterion is present in the first building image or the second building image;

selecting the first building image or the second building image based on the presence of the first minimum criterion in one of the first building image or the second building image;

receiving one or more additional building images comprising imagery of the building object;

aggregating the one or more additional building images with the selected first building image or the second building image based on the presence of a second minimum criterion in the one or more additional building images; and

constructing the multi-dimensional building model of the building object from aggregated building images.

2. The method of claim 1 , wherein the minimum criteria comprises the first minimum criterion and the second minimum criterion.

3. The method of claim 1 , wherein the first minimum criterion and the second minimum criterion are separate minimum criterion.

4. The method of claim 1 , wherein the minimum criteria comprises a plurality of facades.

5. The method of claim 4 , wherein the minimum criteria further comprises architectural features of the plurality of facades.

6. The method of claim 5 , wherein the architectural features comprise roofing or gables.

7. The method of claim 5 , wherein the architectural features comprise windows or doors.

8. The method of claim 1 , wherein the minimum criteria comprises a post of the building object.

9. The method of claim 8 , wherein the post of the building object is a corner.

10. The method of claim 1 , wherein the first building image, the second building image, and the one or more additional building images are captured building images.

11. The method of claim 1 , wherein the first building image, the second building image, and the one or more additional building images are interior building images.

12. The method of claim 11 , wherein the multi-dimensional building model comprises interior facades.

13. The method of claim 1 , wherein the first building image, the second building image, and the one or more additional building images are exterior building images.

14. The method of claim 13 , wherein the multi-dimensional building model comprises exterior facades.

15. The method of claim 1 , wherein the first building image, the second building image, and the one or more additional building images are captured by a portable camera.

16. The method of claim 15 , wherein the portable camera includes any of: a smartphone, a tablet computer, a wearable computer, or a drone.

17. The method of claim 1 , wherein the one or more processors comprise an artificial intelligence system.

18. The method of claim 1 , wherein the multi-dimensional building model comprises a partial multi-dimensional building model.

19. The method of claim 1 , wherein the multi-dimensional building model comprises a base building model.

20. One or more non-transitory computer readable medium storing instructions that, when executed, cause a processor to execute operations comprising:

receiving a request for a multi-dimensional building model based on a minimum criteria, wherein the minimum criteria establishes building images necessary to produce the multi-dimensional building model;

processing a first building image of a building object and a second building image of the building object, wherein processing the first building image and the second building image comprises determining, by one or more processors, if a first minimum criterion is present in the first building image or the second building image;

selecting the first building image or the second building image based on the presence of the first minimum criterion in one of the first building image or the second building image;

receiving one or more additional building images comprising imagery of the building object;

aggregating the one or more additional building images with the selected first building image or the second building image based on the presence of a second minimum criterion in the one or more additional building images; and

constructing the multi-dimensional building model of the building object from aggregated building images.

21. The one or more non-transitory computer readable medium of claim 20 , wherein the minimum criteria comprises the first minimum criterion and the second minimum criterion.

22. The one or more non-transitory computer readable medium of claim 20 , wherein the first minimum criterion and the second minimum criterion are separate minimum criterion.

23. The one or more non-transitory computer readable medium of claim 20 , wherein the minimum criteria comprises a plurality of facades.

24. The one or more non-transitory computer readable medium of claim 23 , wherein the minimum criteria further comprises architectural features of the plurality of facades.

25. The one or more non-transitory computer readable medium of claim 24 , wherein the architectural features comprise roofing or gables.

26. The one or more non-transitory computer readable medium of claim 24 , wherein the architectural features comprise windows or doors.

27. The one or more non-transitory computer readable medium of claim 20 , wherein the minimum criteria comprises a post of the building object.

28. The one or more non-transitory computer readable medium of claim 27 , wherein the post of the building object is a corner.

29. The one or more non-transitory computer readable medium of claim 20 , wherein the first building image, the second building image, and the one or more additional building images are captured building images.

30. The one or more non-transitory computer readable medium of claim 20 , wherein the first building image, the second building image, and the one or more additional building images are interior building images.

31. The one or more non-transitory computer readable medium of claim 30 , wherein the multi-dimensional building model comprises interior facades.

32. The one or more non-transitory computer readable medium of claim 20 , wherein the first building image, the second building image, and the one or more additional building images are exterior building images.

33. The one or more non-transitory computer readable medium of claim 32 , wherein the multi-dimensional building model comprises exterior facades.

34. The one or more non-transitory computer readable medium of claim 20 , wherein the first building image, the second building image, and the one or more additional building images are captured by a portable camera.

35. The one or more non-transitory computer readable medium of claim 34 , wherein the portable camera includes any of: a smartphone, a tablet computer, a wearable computer, or a drone.

36. The one or more non-transitory computer readable medium of claim 20 , wherein the one or more processors comprise an artificial intelligence system.

37. The one or more non-transitory computer readable medium of claim 20 , wherein the multi-dimensional building model comprises a partial multi-dimensional building model.

38. The one or more non-transitory computer readable medium of claim 20 , wherein the multi-dimensional building model comprises a base building model.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 22, 2022
From: UPENDRAN, MANISH; CASTILLO, WILLIAM; MISHRA, AJAY; ALTMAN, ADAM J.
To: HOVER INC.
Reel/Frame 062189/0785 →
Continuity (6)
Continuation 17012835 · Sep 4, 2020
Continuation 16545067 · Aug 20, 2019
Continuation 15942786 · Apr 2, 2018
Continuation In Part 15166587 · May 27, 2016
Provisional Application 62168460 · May 29, 2015
Related Publication 20230092286A1 · Mar 23, 2023
References Cited (70)
US 4798028A · Pinion · 1989 [cited by applicant]
US 5973697A · Berry et al. · 1999 [cited by applicant]
US 7218318B2 · Shimazu · 2007 [cited by applicant]
US 7353114B1 · Rohlf et al. · 2008 [cited by applicant]
US 7814436B2 · Schrag et al. · 2010 [cited by applicant]
US 8040343B2 · Kikuchi et al. · 2011 [cited by applicant]
US 8098899B2 · Ohashi · 2012 [cited by applicant]
US 8139111B2 · Oldroyd · 2012 [cited by applicant]
US 8339394B1 · Lininger · 2012 [cited by applicant]
US 8350850B2 · Steedly et al. · 2013 [cited by applicant]
US 8390617B1 · Reinhardt · 2013 [cited by applicant]
US 8810599B1 · Tseng · 2014 [cited by examiner]
US 9478031B2 · Bhatawadekar et al. · 2016 [cited by applicant]
US 9727834B2 · Reyes · 2017 [cited by applicant]
US 10803658B2 · Upendran · 2020 [cited by examiner]
US 11538219B2 · Upendran · 2022 [cited by examiner]
US 20030014224A1 · Guo et al. · 2003 [cited by applicant]
US 20030052896A1 · Higgins et al. · 2003 [cited by applicant]
US 20040196282A1 · Oh · 2004 [cited by applicant]
US 20060037279A1 · Onchuck · 2006 [cited by applicant]
US 20070168153A1 · Minor et al. · 2007 [cited by applicant]
US 20080112610A1 · Israelsen · 2008 [cited by examiner]
US 20080221843A1 · Shenkar et al. · 2008 [cited by applicant]
US 20080291217A1 · Vincent et al. · 2008 [cited by applicant]
US 20090043504A1 · Bandyopadhyay et al. · 2009 [cited by applicant]
US 20100045869A1 · Baseley et al. · 2010 [cited by applicant]
US 20100074532A1 · Gordon et al. · 2010 [cited by applicant]
US 20100110074A1 · Pershing · 2010 [cited by applicant]
US 20100114537A1 · Pershing · 2010 [cited by applicant]
US 20100214284A1 · Rieffel · 2010 [cited by examiner]
US 20100214291A1 · Muller · 2010 [cited by applicant]
US 20100265048A1 · Lu et al. · 2010 [cited by applicant]
US 20110029897A1 · Russell · 2011 [cited by applicant]
US 20110181589A1 · Quan et al. · 2011 [cited by applicant]
US 20120155778A1 · Buchmueller · 2012 [cited by examiner]
US 20130195362A1 · Janky et al. · 2013 [cited by applicant]
US 20130202157A1 · Pershing · 2013 [cited by applicant]
US 20130222375A1 · Neophytou · 2013 [cited by examiner]
US 20140212028A1 · Ciarcia · 2014 [cited by applicant]
US 20140214473A1 · Gentile et al. · 2014 [cited by applicant]
US 20140247325A1 · Wu et al. · 2014 [cited by applicant]
US 20140278697A1 · Thornberry et al. · 2014 [cited by applicant]
US 20150110385A1 · Schmidt et al. · 2015 [cited by applicant]
US 20150149454A1 · Hieronymus · 2015 [cited by examiner]
US 20150229838A1 · Hakim · 2015 [cited by examiner]
US 20160124435A1 · Thompson · 2016 [cited by examiner]
US 20160350969A1 · Castillo et al. · 2016 [cited by applicant]
WO 2007147830A1 · 2007 [cited by applicant]
WO 2011079241A1 · 2011 [cited by applicant]
WO 2011091552A1 · 2011 [cited by applicant]
Abdul Hasanulhakeem1; A tool to measure dimensions of buildings in various scales for Google Earth Plug-ins and 3D maps; Aug. 6, 2010; pp. 1-2 downloaded from internet: [https://groups.google.com/forum/#!lopic/google-ea… [cited by applicant]
Wang, et al.; Large-Scale Urban Modeling by Combining Ground Level Panoramic and Aerial Imagery; IEEE Third Intemational Symposium on 3D Data Processing, Visualization, and Transmission; Jun. 14-16, 2006; pp. 806-813. [cited by applicant]
Bansal, et al., “Geo-Localization of Street Views with Aerial Image Databases,” Nov. 28-Dec. 1, 2011, pp. 1125-1128. [cited by applicant]
Becker, et al., “Semiautomatic 3-D model extraction from uncalibrated 2-D camera views,” MIT Media Laboratory, Apr. 1995, 15 pages. [cited by applicant]
Caramba App Development, “EasyMeasure—Measure with your Camera on the App Store on iTunes”, https://tunes.apple.com/us/app/easymeasure-measure-measure-your-camera/id349530105mt=8, 2010, 2 pages. [cited by applicant]
Chen, et al., “City-Scale Landmark Identification on Mobile Devices,” Jul. 2011, pp. 737-744. [cited by applicant]
Fairfax County Virginia, “Virtual Fairfax,” http://www.fairfaxcounty.gov/gis/virtualfairfax, Feb. 24, 2014; 2 pages. [cited by applicant]
Fruh and Zakhor, “Constructing 3D City Models by Merging Aerial and Ground Views,” IEEE Computer Graphics and Applications, Nov./Dec. 2003, pp. 52-61, 10 pages. [cited by applicant]
Huang and Wu, et al., “Towards 3D City Modeling through Combining Ground Level Panoramic and Orthogonal Aerial Imagery,” 2011 Workshop on Digital Media and Digital Content Management, pp. 66-71, 6 pages. [cited by applicant]
Jaynes, “View Alignment of Aerial and Terrestrial Imagery in Urban Environments,” Springer-Verlag Berlin Heidelberg 1999, pp. 3-19, 17 pages. [cited by applicant]
Kroepfl, et al., “Efficiently Locating Photographs in Many Panoramas,” Nov. 2-5, 2010, ACM GIS10. [cited by applicant]
Lee, et al., “Automatic Integration of Facade Textures into 3D Building Models with a Projective Geometry Based Line Clustering,” Eurographics 2002, vol. 2, No. 3, 10 pages. [cited by applicant]
Lee, et al., “Integrating Ground and Aerial Views for Urban Site Modeling,” 2002; 6 pages. [cited by applicant]
Pu et al., “Automatic Extraction of Building Features From Terrestrial Laser Scanning,” 2006, International Institute for Geo-information Science and Earth Observation, 5 pages. [cited by applicant]
Scale & Area Measurement; (n.d). Retrieved from http://www.geog.ucsb.edu/˜jeff/115a/lectures/scale_and_area_measurement.html, 8 pages. [cited by applicant]
Scope Technologies; Solutions; Mar. 4, 2014; pp. 1-2 downloaded from the internet: [http://www.myscopetech.com/solutions.php]. [cited by applicant]
Xiao, et al., “Image-based Facade Modeling,” ACM Transaction on Graphics (TOG), 2008, 10 pages. [cited by applicant]
SketchUp Knowledge Base, Tape Measure Tool: Scaling an entire model, http://help.sketchup.com/en/article/95006, 2013 Trimble Navigation Limited, 2 pages. [cited by applicant]
Ou et al., “A New Method for Automatic Large Scale Map Updating Using Mobile Mapping Magery”, Sep. 2013, Wiley Online Library, vol. 28, No. 143, pp. 240-260. [cited by applicant]
Vosselman et al., “3D Building Model Reconstruction From Point Clouds and Ground Plans”, Oct. 2001, Natural Resources Canada, vol. 34, No. 3/W4, pp. 37-44. [cited by applicant]