IP Library Granted Patent US 12,400,049
Granted Patent B2
US 12,400,049 · App. 18/236,728 · Granted Aug 26, 2025

System and method for generating computerized models of structures using geometry extraction and reconstruction techniques

Inventors: Jeffery D. Lewis (Orem, UT); Jeffrey C. Taylor (Alpine, UT)
Assignee: Xactware Solutions, Inc.
G06F30/13G06T7/12G06T7/55G06T2207/10028
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,400,049
App. No.
18/236,728
Granted
Aug 26, 2025
Kind
B2
Abstract

Described in detail herein are systems and methods for generating computerized models of structures using geometry extraction and reconstruction techniques. The system includes a computing device coupled to a input device. The input device obtains raw data scanned by a sensor. The computing device is programmed to execute a data fusion process is applied to fuse the raw data, and a geometry extraction process is performed on the fused data to extract features such as walls, floors, ceilings, roof planes, etc. Large- and small-scale features of the structure are reconstructed using the extracted features. The large- and small-scale features are reconstructed by the system into a floor plan (contour) and/or a polyhedron corresponding to the structure. The system can also process exterior features of the structure to automatically identify condition and areas of roof damage.

Claims (293)

1. A system for generating computerized models of structures, the system comprising:

an input device including one or more sensors, the input device configured to:

capture, via the one or more sensors, raw data of at least one structure disposed in a physical space and including a plurality of large-scale structure features and a plurality of small-scale structure features, wherein the large-scale structure features and small-scale structure features include geometric features;

a computing system coupled to the input device and including, a processor, a display, and a memory, the computing system programmed to:

receive the raw data;

process the raw data using a data fusion process to produce fused data;

extract the geometric features from the fused data;

reconstruct the plurality of large-scale structure features using the extracted geometric features to generate a floor contour or polyhedron representing the plurality of large-scale structure features;

reconstruct the small-scale structure features within the floor contour or polyhedron using the extracted geometric features;

store the large-scale structure features and small-scale features;

generate a three-dimensional visual representation of a reconstructed model of the at least one structure using the large-scale structure features and small-scale features; and

display, on the display, the three-dimensional visual representation of the reconstructed model of the at least one structure;

wherein the raw data comprises one or more of: RGB image data, infrared image data, mobile sensors data, point cloud data, LIDAR data, global positioning system (GPS) data, X-ray data, magnetic field data, and depth maps data; and

wherein the computing system is programmed to:

 detect the type of raw data is point cloud data, in response to receiving the raw data;

 identify one or more major planes in the raw data;

 flatten the one or more planes by projecting points of the one or more plane onto the one or more planes;

 identify at least one floor plane and at least one wall plane;

 identify one or more corner points formed by the joining of the at least one floor plane and at least one wall plane;

 generate a fine mesh horizontal grid including a plurality of cells;

 assign values to the plurality of cells in the grid;

 identify one or more floor plans including a plurality of edges, based on the identified one or more corner points; and

 assign a score to each of the one or more floor plans, wherein the score is calculated based on a comparison of the plurality of edges and the plurality of the cells in the grid.

2. The system of claim 1 , wherein the geometry features include one or more of: edges lines, planes, points and corners.

3. The system of claim 1 , wherein the large-scale structure features include one or more of: walls, ceilings and floors.

4. The system of claim 1 , wherein the computing system reconstructs the large-scale structure features using the extracted geometry features by:

identifying one or more of the extracted geometry features corresponding a ceiling and floor of the at least one structure;

identifying one or more of the extracted geometry features corresponding to one or more walls of the at least one structure;

straightening the one or more walls;

identifying adjacent walls from the one or more walls; and

squaring an angle formed by adjacent walls.

5. The system of claim 1 , wherein the small-scale structure features include one or more of: room types, materials, wall features and fixtures.

6. The system of claim 1 , wherein the computing system is configured to identify the at least one floor plane and the at least one wall plane based on a user interaction with the display.

7. The system of claim 1 , further comprising an image capturing device coupled to the input device, the image capturing device configured to:

capture one or more images of a roof face of the at least one structure; and

transmit the one or more images of the roof face of the at least one structure.

8. The system in claim 7 , wherein in response to displaying on the display the reconstructed model of the at least one structure, the computing system further programmed to:

receive the one or more images of the roof face of the at least one structure;

determine a pixel size of the one or more images;

map the one or more images to the reconstructed model of the at least one structure;

identify one or more areas of damage in the one or more images; and

determine an amount of the one or more areas of damage exceeds a predetermined threshold amount.

9. The system of claim 1 , wherein the computing system is one of: a mobile device or a remote structure modeling server.

10. The system of claim 1 , wherein the one or more sensors are one or more of:

image capturing device, laser scanners, scanning range finders, RGB-D sensors, mobile sensors.

11. The system of claim 1 , wherein the input device is one of: smart phones, tablet computers, laptop computers, wearable computers, robots or Unmanned Aerial Vehicle (UAVs).

12. A method for generating computerized models of structures, the method comprising:

capturing, via an input device including one or more sensors, using the one or more sensors, raw data of at least one structure disposed in physical space and including a plurality of large-scale structure features and a plurality of small-scale structure features, wherein the large-scale structure features and small-scale structure features include geometric features, and wherein the raw data comprises one or more of: RGB image data, infrared image data, mobile sensors data, point cloud data, LIDAR data, global positioning system (GPS) data, X-ray data, magnetic field data, and depth maps data;

transmitting, via the input device, the raw data;

receiving, via the computing system, the raw data;

processing the raw data using a data fusion process to produce fused data;

storing, via the computing system, the fused data;

extracting, via the computing system, the geometric features from the fused data;

reconstructing, via the computing system, the plurality of large-scale structure features using the extracted geometric features to generate a floor contour or polyhedron representing the plurality of large-scale structure features;

reconstructing, via the computing system, the small-scale structure features within the floor contour or polyhedron using the extracted geometric features;

storing, via the computing system, the large-scale structure features and small-scale features;

generating, via the computing system, a three-dimensional visual representation of a reconstructed model of the at least one structure using the large-scale structure features and small-scale features; and

displaying, via the computing system, on the display the three-dimensional visual representation of the reconstructed model the reconstructed model of the at least one structure;

detecting, via the computing system, the type of raw data is point cloud data, in response to receiving the raw data;

identifying, via the computing system, one or more major planes in the raw data;

flattening, via the computing system, the one or more planes by projecting points of the one or more plane onto the one or more planes;

identifying, via the computing system, at least one floor plane and at least one wall plane;

identifying, via the computing system, one or more corner points formed by the joining of the at least one floor plane and at least one wall plane;

generating, via the computing system, a fine mesh horizontal grid including a plurality of cells;

assigning, via the computing system, values to the plurality of cells in the grid;

identifying, via the computing system, one or more floor plans including a plurality of edges, based on the identified one or more corner points; and

assigning, via the computing system, a score to each of the one or more floor plans, wherein the score is calculated based on a comparison of the plurality of edges and the plurality of the cells in the grid.

13. The method of claim 12 , wherein reconstructing the large-scale structure features using the extracted geometry features further comprising:

identifying, via the computing system, one or more of the extracted geometry features corresponding a ceiling and a floor of the at least one structure;

identifying, via the computing system, one or more of the extracted geometry features corresponding to one or more walls of the at least one structure;

straightening, via the computing system, the one or more walls;

identifying, via the computing system, adjacent walls from the one or more walls; and

squaring, via the computing system, an angle formed by adjacent walls.

14. The method of claim 12 , further comprising:

capturing, via an image capturing device coupled to the input device, one or more images of a roof face of the at least one structure; and

transmitting, via the input device, the one or more images of the roof face of the at least one structure.

15. The method in claim 14 , wherein in response to displaying on the display the reconstructed model of the at least one structure, the method further comprising:

receiving, via the computing system, the one or more images of the roof face of the at least one structure;

determining, via the computing system, a pixel size of the one or more images;

mapping, via the computing system, the one or more images to the reconstructed model of the at least one structure;

identifying, via the computing system, one or more areas of damage in the one or more images; and

determining, via the computing system, an amount of the one or more areas of damage exceeds a predetermined threshold amount.

16. A non-transitory computer readable medium storing computer executable instructions thereupon for generating computerized models of structures, the instructions when executed by a processor cause the processor to:

capture, via an input device including one or more sensors, using the one or more sensors, raw data of at least one structure disposed in a physical space and including a plurality of large-scale structure features and a plurality of small-scale structure features, wherein the large-scale structure features and small-scale structure features include geometric features;

transmit, via the input device, the raw data;

receive, via the computing system, the raw data;

processing the raw data using a data fusion process to produce fused data;

store, via the computing system, the fused data;

extract, via the computing system, the geometry features from the fused data;

reconstruct, via the computing system, the plurality of large-scale structure features using the extracted geometry features to generate a floor contour or polyhedron representing the plurality of large-scale structure features;

reconstruct, via the computing system, the small-scale structure features within the floor contour or the polyhedon using the extracted geometry features;

store, via the computing system, the large-scale structure features and small-scale features;

generate, via the computing system, a three-dimensional visual representation of a reconstructed model of the at least one structure using the large-scale structure features and small-scale features; and

display, via the computing system, on the display the three-dimensional visual representation of the reconstructed model of the at least one structure;

wherein the raw data comprises one or more of: RGB image data, infrared image data, mobile sensors data, point cloud data, LIDAR data, global positioning system (GPS) data, X-ray data, magnetic field data, and depth maps data; and

wherein the instructions, when executed by the processor, further cause the processor to:

detect, via the computing system, the type of raw data is point cloud data, in response to receiving the raw data;

calculate, via the computing system, one or more raw lines from the point cloud data;

calculate, via the computing system, main guidelines based on the one or more raw lines;

calculate, via the computing system, a pivot point from one or more intersection points of the main guidelines;

identify, via the computing system, an orthogonal facade for the at least one structure;

identify, via the computing system, one or more rooms of the at least one structure based on the main guidelines, the pivot point and orthogonal façade;

generate, via the computing system, a visual representation of the one or more rooms of the at least on structure; and

display, via the computing system, on the display the visual representation of the one or more rooms of the at least one structure.

17. The non-transitory computer readable medium of claim 16 , wherein reconstructing the large-scale structure features using the extracted geometry features, when executed by the processor, further cause the processor to:

identify, via the computing system, one or more of the extracted geometry features corresponding a ceiling and a floor of the at least one structure;

identify, via the computing system, one or more of the extracted geometry features corresponding to one or more walls of the at least one structure;

straighten, via the computing system, the one or more walls;

identify, via the computing system, adjacent walls from the one or more walls; and

square, via the computing system, an angle formed by adjacent walls.

18. The non-transitory computer readable medium of claim 16 , wherein the instructions, when executed by the processor, further cause the processor to:

capture, via an image capturing device coupled to the input device, one or more images of a roof face of the at least one structure; and

transmit, via the input device, the one or more images of the roof face of the at least one structure.

19. The non-transitory computer readable medium of claim 18 , wherein the instructions, when executed by the processor, further cause the processor to:

receive, via the computing system, the one or more images of the roof face of the at least one structure;

determine, via the computing system, a pixel size of the one or more images;

map, via the computing system, the one or more images to the reconstructed model of the at least one structure;

identify, via the computing system, one or more areas of damage in the one or more images; and

determine, via the computing system, an amount of the one or more areas of damage exceeds a predetermined threshold amount.

20. A system for generating computerized models of structures, the system comprising:

an input device including one or more sensors, the input device configured to:

capture, via the one or more sensors, raw data of at least one structure disposed in a physical space and including a plurality of large-scale structure features and a plurality of small-scale structure features, wherein the large-scale structure features and small-scale structure features include geometric features;

a computing system coupled to the input device and including, a processor, a display, and a memory, the computing system programmed to:

receive the raw data;

process the raw data using a data fusion process to produce fused data;

extract the geometric features from the fused data;

reconstruct the plurality of large-scale structure features using the extracted geometric features to generate a floor contour or polyhedron representing the plurality of large-scale structure features;

reconstruct the small-scale structure features within the floor contour or polyhedron using the extracted geometric features;

store the large-scale structure features and small-scale features;

generate a three-dimensional visual representation of a reconstructed model of the at least one structure using the large-scale structure features and small-scale features; and

display, on the display, the three-dimensional visual representation of the reconstructed model of the at least one structure;

wherein the raw data comprises one or more of: RGB image data, infrared image data, mobile sensors data, point cloud data, LIDAR data, global positioning system (GPS) data, X-ray data, magnetic field data, and depth maps data; and

wherein the computing system is programmed to:

detect the type of raw data is point cloud data, in response to receiving the raw data;

calculate one or more raw lines from the point cloud data;

calculate main guidelines based on the one or more raw lines;

calculate a pivot point from one or more intersection points of the main guidelines;

identify an orthogonal façade for the at least one structure;

identify one or more rooms of the at least one structure based on the main guidelines, the pivot point and orthogonal façade;

generate a visual representation of the one or more rooms of the at least on structure; and

display on the display the visual representation of the one or more rooms of the at least one structure.

21. A system for generating computerized models of structures, the system comprising:

an input device including one or more sensors, the input device configured to:

capture, via the one or more sensors, raw data of at least one structure disposed in a physical space and including a plurality of large-scale structure features and a plurality of small-scale structure features, wherein the large-scale structure features and small-scale structure features include geometric features;

a computing system coupled to the input device and including, a processor, a display, and a memory, the computing system programmed to:

receive the raw data;

process the raw data using a data fusion process to produce fused data;

extract the geometric features from the fused data;

reconstruct the plurality of large-scale structure features using the extracted geometric features to generate a floor contour or polyhedron representing the plurality of large-scale structure features;

reconstruct the small-scale structure features within the floor contour or polyhedron using the extracted geometric features;

store the large-scale structure features and small-scale features;

generate a three-dimensional visual representation of a reconstructed model of the at least one structure using the large-scale structure features and small-scale features; and

display, on the display, the three-dimensional visual representation of the reconstructed model of the at least one structure;

wherein the raw data comprises one or more of: RGB image data, infrared image data, mobile sensors data, point cloud data, LIDAR data, global positioning system (GPS) data, X-ray data, magnetic field data, and depth maps data; and

wherein the computing system is programmed to:

detect the type of raw data is point cloud data, in response to receiving the raw data;

extract a main axis from the point cloud data;

create a fine mesh horizontal grid including a plurality of cells;

assign a value to each of the plurality of cells;

calculate main guidelines;

calculate corners based on one or more intersection points of one or more lines;

identify one or more walls of the least one structure based on the calculated corners;

generate a first floor plan of the at least one structure;

compare a second floor plan to the first floor plan of the at least one structure; and

assign a score to the second floor plan based on the comparison of the second floor plan and the first plan of the at least one structure.

22. A system for generating computerized models of structures, the system comprising:

an input device including one or more sensors, the input device configured to:

capture, via the one or more sensors, raw data of at least one structure disposed in a physical space and including a plurality of large-scale structure features and a plurality of small-scale structure features, wherein the large-scale structure features and small-scale structure features include geometric features;

a computing system coupled to the input device and including, a processor, a display, and a memory, the computing system programmed to:

receive the raw data;

process the raw data using a data fusion process to produce fused data;

extract the geometric features from the fused data;

reconstruct the plurality of large-scale structure features using the extracted geometric features to generate a floor contour or polyhedron representing the plurality of large-scale structure features;

reconstruct the small-scale structure features within the floor contour or polyhedron using the extracted geometric features;

store the large-scale structure features and small-scale features;

generate a three-dimensional visual representation of a reconstructed model of the at least one structure using the large-scale structure features and small-scale features; and

display, on the display, the three-dimensional visual representation of the reconstructed model of the at least one structure;

wherein the raw data comprises one or more of: RGB image data, infrared image data, mobile sensors data, point cloud data, LIDAR data, global positioning system (GPS) data, X-ray data, magnetic field data, and depth maps data; and

wherein the computing system is programmed to:

detect the type of raw data is point cloud data captured at a first-person perspective view, in response to receiving the raw data;

receiving a user interaction on the display identifying at least one wall of the at least one structure in the point cloud data;

project an infinite ray into the point cloud data;

determine one or more intersection points of the infinite ray and the point cloud data;

execute a region growing plane algorithm based on the one or more intersection points of the infinite ray and the point cloud data;

extract at least one plane of the at least one wall in response to the execution of the region growing plane algorithm;

calculate one or more intersection points of the at least one wall and at least one floor or at least one ceiling of the at least one structure; and

display, on the display, the floor contour or the polyhedron.

23. A method for generating computerized models of structures, the method comprising:

capturing, via an input device including one or more sensors, using the one or more sensors, raw data of at least one structure disposed in physical space and including a plurality of large-scale structure features and a plurality of small-scale structure features, wherein the large-scale structure features and small-scale structure features include geometric features, and wherein the raw data comprises one or more of: RGB image data, infrared image data, mobile sensors data, point cloud data, LIDAR data, global positioning system (GPS) data, X-ray data, magnetic field data, and depth maps data;

transmitting, via the input device, the raw data;

receiving, via the computing system, the raw data;

processing the raw data using a data fusion process to produce fused data;

storing, via the computing system, the fused data;

extracting, via the computing system, the geometric features from the fused data;

reconstructing, via the computing system, the plurality of large-scale structure features using the extracted geometric features to generate a floor contour or polyhedron representing the plurality of large-scale structure features;

reconstructing, via the computing system, the small-scale structure features within the floor contour or polyhedron using the extracted geometric features;

storing, via the computing system, the large-scale structure features and small-scale features;

generating, via the computing system, a three-dimensional visual representation of a reconstructed model of the at least one structure using the large-scale structure features and small-scale features;

displaying, via the computing system, on the display the three-dimensional visual representation of the reconstructed model the reconstructed model of the at least one structure;

detecting, via the computing system, the type of raw data is point cloud data, in response to receiving the raw data;

calculating, via the computing system, one or more raw lines from the point cloud data;

calculating, via the computing system, main guidelines based on the one or more raw lines;

calculating, via the computing system, a pivot point from one or more intersection points of the main guidelines;

identifying, via the computing system, an orthogonal façade for the at least one structure;

identifying, via the computing system, one or more rooms of the at least one structure based on the main guidelines, the pivot point and orthogonal façade;

generating, via the computing system, a visual representation of the one or more rooms of the at least on structure; and

displaying, via the computing system, on the display the visual representation of the one or more rooms of the at least one structure.

24. A method for generating computerized models of structures, the method comprising:

capturing, via an input device including one or more sensors, using the one or more sensors, raw data of at least one structure disposed in physical space and including a plurality of large-scale structure features and a plurality of small-scale structure features, wherein the large-scale structure features and small-scale structure features include geometric features, and wherein the raw data comprises one or more of: RGB image data, infrared image data, mobile sensors data, point cloud data, LIDAR data, global positioning system (GPS) data, X-ray data, magnetic field data, and depth maps data;

transmitting, via the input device, the raw data;

receiving, via the computing system, the raw data;

processing the raw data using a data fusion process to produce fused data;

storing, via the computing system, the fused data;

extracting, via the computing system, the geometric features from the fused data;

reconstructing, via the computing system, the plurality of large-scale structure features using the extracted geometric features to generate a floor contour or polyhedron representing the plurality of large-scale structure features;

reconstructing, via the computing system, the small-scale structure features within the floor contour or polyhedron using the extracted geometric features;

storing, via the computing system, the large-scale structure features and small-scale features;

generating, via the computing system, a three-dimensional visual representation of a reconstructed model of the at least one structure using the large-scale structure features and small-scale features;

displaying, via the computing system, on the display the three-dimensional visual representation of the reconstructed model the reconstructed model of the at least one structure;

detecting, via the computing system, the type of raw data is point cloud data, in response to receiving the raw data;

extracting, via the computing system, a main axis from the point cloud data;

creating, via the computing system, a fine mesh horizontal grid including a plurality of cells;

assigning, via the computing system, a value to each of the plurality of cells;

calculating, via the computing system, main guidelines;

calculating, via the computing system, corners based on one or more intersection points of one or more lines;

identifying, via the computing system, one or more walls of the least one structure based on the calculated corners;

generating, via the computing system, a first floor plan of the at least one structure;

comparing, via the computing system, a second floor plan to the first floor plan of the at least one structure; and

assigning, via the computing system, a score to the second floor plan based on the comparison of the second floor plan and the first plan of the at least one structure.

25. A method for generating computerized models of structures, the method comprising:

capturing, via an input device including one or more sensors, using the one or more sensors, raw data of at least one structure disposed in physical space and including a plurality of large-scale structure features and a plurality of small-scale structure features, wherein the large-scale structure features and small-scale structure features include geometric features, and wherein the raw data comprises one or more of: RGB image data, infrared image data, mobile sensors data, point cloud data, LIDAR data, global positioning system (GPS) data, X-ray data, magnetic field data, and depth maps data;

transmitting, via the input device, the raw data;

receiving, via the computing system, the raw data;

processing the raw data using a data fusion process to produce fused data;

storing, via the computing system, the fused data;

extracting, via the computing system, the geometric features from the fused data;

reconstructing, via the computing system, the plurality of large-scale structure features using the extracted geometric features to generate a floor contour or polyhedron representing the plurality of large-scale structure features;

reconstructing, via the computing system, the small-scale structure features within the floor contour or polyhedron using the extracted geometric features;

storing, via the computing system, the large-scale structure features and small-scale features;

generating, via the computing system, a three-dimensional visual representation of a reconstructed model of the at least one structure using the large-scale structure features and small-scale features;

displaying, via the computing system, on the display the three-dimensional visual representation of the reconstructed model the reconstructed model of the at least one structure;

detecting, via the computing system, the type of raw data is point cloud data captured at a first-person perspective view, in response to receiving the raw data;

receiving, via the computing system, a user interaction on the display identifying at least one wall of the at least one structure in the point cloud data;

projecting, via the computing system, an infinite ray into the point cloud data;

determining, via the computing system, one or more intersection points of the infinite ray and the point cloud data;

executing, via the computing system, a region growing plane algorithm based on the one or more intersection points of the infinite ray and the point cloud data;

extracting, via the computing system, at least one plane of the at least one wall in response to the execution of the region growing plane algorithm;

calculating, via the computing system, one or more intersection points of the at least one wall and at least one floor or at least one ceiling of the at least one structure; and

displaying, via the computing system, on the display, the floor contour or the polyhedron.

26. A non-transitory computer readable medium storing computer executable instructions thereupon for generating computerized models of structures, the instructions when executed by a processor cause the processor to:

capture, via an input device including one or more sensors, using the one or more sensors, raw data of at least one structure disposed in a physical space and including a plurality of large-scale structure features and a plurality of small-scale structure features, wherein the large-scale structure features and small-scale structure features include geometric features, and wherein the raw data comprises one or more of: RGB image data, infrared image data, mobile sensors data, point cloud data, LIDAR data, global positioning system (GPS) data, X-ray data, magnetic field data, and depth maps data;

transmit, via the input device, the raw data;

receive, via the computing system, the raw data;

processing the raw data using a data fusion process to produce fused data;

store, via the computing system, the fused data;

extract, via the computing system, the geometry features from the fused data;

reconstruct, via the computing system, the plurality of large-scale structure features using the extracted geometry features to generate a floor contour or polyhedron representing the plurality of large-scale structure features;

reconstruct, via the computing system, the small-scale structure features within the floor contour or the polyhedon using the extracted geometry features;

store, via the computing system, the large-scale structure features and small-scale features;

generate, via the computing system, a three-dimensional visual representation of a reconstructed model of the at least one structure using the large-scale structure features and small-scale features;

display, via the computing system, on the display the three-dimensional visual representation of the reconstructed model of the at least one structure;

wherein the instructions, when executed by the processor, further cause the processor to:

detect, via the computing system, the type of raw data is point cloud data, in response to receiving the raw data;

extract, via the computing system, a main axis from the point cloud data;

create, via the computing system, a fine mesh horizontal grid including a plurality of cells;

assign, via the computing system, a value to each of the plurality of cells;

calculate, via the computing system, main guidelines;

calculate, via the computing system, corners based on one or more intersection points of one or more lines;

identify, via the computing system, one or more walls of the least one structure based on the calculated corners;

generate, via the computing system, a first floor plan of the at least one structure;

compare, via the computing system, a second floor plan to the first floor plan of the at least one structure; and

assign, via the computing system, a score to the second floor plan based on the comparison of the second floor plan and the first plan of the at least one structure.

27. A non-transitory computer readable medium storing computer executable instructions thereupon for generating computerized models of structures, the instructions when executed by a processor cause the processor to:

capture, via an input device including one or more sensors, using the one or more sensors, raw data of at least one structure disposed in a physical space and including a plurality of large-scale structure features and a plurality of small-scale structure features, wherein the large-scale structure features and small-scale structure features include geometric features, and wherein the raw data comprises one or more of: RGB image data, infrared image data, mobile sensors data, point cloud data, LIDAR data, global positioning system (GPS) data, X-ray data, magnetic field data, and depth maps data;

transmit, via the input device, the raw data;

receive, via the computing system, the raw data;

processing the raw data using a data fusion process to produce fused data;

store, via the computing system, the fused data;

extract, via the computing system, the geometry features from the fused data;

reconstruct, via the computing system, the plurality of large-scale structure features using the extracted geometry features to generate a floor contour or polyhedron representing the plurality of large-scale structure features;

reconstruct, via the computing system, the small-scale structure features within the floor contour or the polyhedon using the extracted geometry features;

store, via the computing system, the large-scale structure features and small-scale features;

generate, via the computing system, a three-dimensional visual representation of a reconstructed model of the at least one structure using the large-scale structure features and small-scale features;

display, via the computing system, on the display the three-dimensional visual representation of the reconstructed model of the at least one structure;

wherein the instructions, when executed by the processor, further cause the processor to:

detect, via the computing system, the type of raw data is point cloud data captured at a first-person perspective view, in response to receiving the raw data;

receive, via the computing system, a user interaction on the display identifying at least one wall of the at least one structure in the point cloud data;

project, via the computing system, an infinite ray into the point cloud data;

determine, via the computing system, one or more intersection points of the infinite ray and the point cloud data;

execute, via the computing system, a region growing plane algorithm based on the one or more intersection points of the infinite ray and the point cloud data;

extract, via the computing system, at least one plane of the at least one wall in response to the execution of the region growing plane algorithm;

calculate, via the computing system, one or more intersection points of the at least one wall and at least one floor or at least one ceiling of the at least one structure; and

display, via the computing system, on the display, the floor contour or the polyhedron.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 22, 2023
From: LEWIS, JEFFERY D.; TAYLOR, JEFFREY C.
To: GEOMNI, INC.
Reel/Frame 064670/0377 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 22, 2023
From: GEOMNI, INC.
To: XACTWARE SOLUTIONS, INC.
Reel/Frame 064670/0380 →
Continuity (4)
Continuation 16545607 · Aug 20, 2019
Continuation 15374695 · Dec 9, 2016
Provisional Application 62265359 · Dec 9, 2015
Related Publication 20230409769A1 · Dec 21, 2023
References Cited (173)
US 5701403A · Watanabe et al. · 1997 [cited by applicant]
US 6446030B1 · Hoffman et al. · 2002 [cited by applicant]
US 6448964B1 · Isaacs et al. · 2002 [cited by applicant]
US 8533063B2 · Erickson · 2013 [cited by applicant]
US 8843304B1 · Dupont et al. · 2014 [cited by applicant]
US 8868375B1 · Christian · 2014 [cited by applicant]
US 8983806B2 · Labrie et al. · 2015 [cited by applicant]
US 9158869B2 · Labrie et al. · 2015 [cited by applicant]
US 9430871B2 · Neophytou et al. · 2016 [cited by applicant]
US 9501700B2 · Loveland et al. · 2016 [cited by applicant]
US 9679227B2 · Taylor et al. · 2017 [cited by applicant]
US 10127670B2 · Lewis et al. · 2018 [cited by applicant]
US 10181079B2 · Labrie et al. · 2019 [cited by applicant]
US 10289760B1 · Oakes, III et al. · 2019 [cited by applicant]
US 10387582B2 · Lewis et al. · 2019 [cited by applicant]
US 10445438B1 · Motonaga et al. · 2019 [cited by applicant]
US 10529028B1 · Davis et al. · 2020 [cited by applicant]
US 11314905B2 · Childs et al. · 2022 [cited by applicant]
US 11688135B2 · Lewis et al. · 2023 [cited by applicant]
US 11688186B2 · Harris et al. · 2023 [cited by applicant]
US 11734468B2 · Lewis et al. · 2023 [cited by applicant]
US 12124775B2 · Childs et al. · 2024 [cited by applicant]
US 12125139B2 · Lewis et al. · 2024 [cited by applicant]
US 20020116254A1 · Stein et al. · 2002 [cited by applicant]
US 20030009315A1 · Thomas et al. · 2003 [cited by applicant]
US 20070080961A1 · Inzinga et al. · 2007 [cited by applicant]
US 20070276626A1 · Bruffey · 2007 [cited by applicant]
US 20090179895A1 · Zhu et al. · 2009 [cited by applicant]
US 20100110074A1 · Pershing · 2010 [cited by applicant]
US 20100114537A1 · Pershing · 2010 [cited by applicant]
US 20100296693A1 · Thornberry et al. · 2010 [cited by applicant]
US 20110056286A1 · Jansen · 2011 [cited by applicant]
US 20110157213A1 · Takeyama et al. · 2011 [cited by applicant]
US 20110191738A1 · Walker et al. · 2011 [cited by applicant]
US 20120026322A1 · Malka et al. · 2012 [cited by applicant]
US 20120179431A1 · Labrie et al. · 2012 [cited by applicant]
US 20120253725A1 · Malka et al. · 2012 [cited by applicant]
US 20120253751A1 · Malka et al. · 2012 [cited by applicant]
US 20130201167A1 · Oh et al. · 2013 [cited by applicant]
US 20130206177A1 · Burlutskiy · 2013 [cited by applicant]
US 20130226451A1 · O'Neill et al. · 2013 [cited by applicant]
US 20130262029A1 · Pershing · 2013 [cited by applicant]
US 20130267260A1 · Chao et al. · 2013 [cited by applicant]
US 20130314688A1 · Likholyot · 2013 [cited by applicant]
US 20140043436A1 · Bell et al. · 2014 [cited by applicant]
US 20140195275A1 · Pershing et al. · 2014 [cited by applicant]
US 20140301633A1 · Furukawa et al. · 2014 [cited by applicant]
US 20140320661A1 · Sankar et al. · 2014 [cited by applicant]
US 20150029182A1 · Sun et al. · 2015 [cited by applicant]
US 20150073864A1 · Labrie et al. · 2015 [cited by applicant]
US 20150081252A1 · Loss · 2015 [cited by examiner]
US 20150093047A1 · Battcher et al. · 2015 [cited by applicant]
US 20150116509A1 · Birkler et al. · 2015 [cited by applicant]
US 20150153172A1 · Starns et al. · 2015 [cited by applicant]
US 20150193971A1 · Dryanovski et al. · 2015 [cited by applicant]
US 20150213558A1 · Nelson · 2015 [cited by applicant]
US 20150227645A1 · Childs et al. · 2015 [cited by applicant]
US 20150269438A1 · Samarasekera et al. · 2015 [cited by applicant]
US 20150302529A1 · Jagannathan · 2015 [cited by applicant]
US 20160098802A1 · Bruffey et al. · 2016 [cited by applicant]
US 20160110480A1 · Randolph · 2016 [cited by applicant]
US 20160246767A1 · Makadia et al. · 2016 [cited by applicant]
US 20160282107A1 · Roland et al. · 2016 [cited by applicant]
US 20170124713A1 · Jurgenson et al. · 2017 [cited by applicant]
US 20170132711A1 · Bruffey et al. · 2017 [cited by applicant]
US 20170132835A1 · Halliday et al. · 2017 [cited by applicant]
US 20170169459A1 · Bruffey et al. · 2017 [cited by applicant]
US 20170193297A1 · Michini et al. · 2017 [cited by applicant]
US 20170206648A1 · Marra et al. · 2017 [cited by applicant]
US 20170221152A1 · Nelson et al. · 2017 [cited by applicant]
US 20170316115A1 · Lewis et al. · 2017 [cited by applicant]
US 20170330207A1 · Labrie et al. · 2017 [cited by applicant]
US 20170345069A1 · Labrie et al. · 2017 [cited by applicant]
US 20180053329A1 · Roberts et al. · 2018 [cited by applicant]
US 20180067593A1 · Tiwari et al. · 2018 [cited by applicant]
US 20180089833A1 · Lewis et al. · 2018 [cited by applicant]
US 20180286098A1 · Lorenzo · 2018 [cited by applicant]
US 20180330528A1 · Loveland et al. · 2018 [cited by applicant]
US 20180357819A1 · Oprea · 2018 [cited by applicant]
US 20180373931A1 · Li · 2018 [cited by applicant]
US 20190114717A1 · Labrie et al. · 2019 [cited by applicant]
US 20190147247A1 · Harris et al. · 2019 [cited by applicant]
US 20190221040A1 · Shantharam et al. · 2019 [cited by applicant]
US 20190340692A1 · Labrie et al. · 2019 [cited by applicant]
US 20190347859A1 · Jovanovic et al. · 2019 [cited by applicant]
US 20190377837A1 · Lewis et al. · 2019 [cited by applicant]
US 20200100066A1 · Lewis et al. · 2020 [cited by applicant]
US 20200143481A1 · Brown et al. · 2020 [cited by applicant]
US 20210076162A1 · Wang et al. · 2021 [cited by applicant]
US 20210103687A1 · Harris et al. · 2021 [cited by applicant]
US 20210350038A1 · Jenson et al. · 2021 [cited by applicant]
US 20220309204A1 · Childs et al. · 2022 [cited by applicant]
US 20220309748A1 · Lewis et al. · 2022 [cited by applicant]
US 20220414980A1 · Lewis et al. · 2022 [cited by applicant]
US 20230343030A1 · Lewis et al. · 2023 [cited by applicant]
WO 2014151122A1 · 2014 [cited by applicant]
WO 2016154306A1 · 2016 [cited by applicant]
WO 2017100658A1 · 2017 [cited by applicant]
Office Action mailed Oct. 13, 2022, issued in connection with U.S. Appl. No. 16/545,607 (53 pages). [cited by applicant]
International Search Report of the International Searching Authority mailed on Nov. 18, 2022, issued in connection with International Application No. PCT/US22/030691 (6 pages). [cited by applicant]
Written Opinion of the International Searching Authority mailed on Nov. 18, 2022, issued in connection with International Application No. PCT/US22/030691 (11 pages). [cited by applicant]
Notice of Allowance mailed Dec. 9, 2022, issued in connection with U.S. Appl. No. 17/705,130 (10 pages). [cited by applicant]
Examination Report No. 1 mailed Dec. 15, 2022, issued by the Australian Patent Office in connection with Australian Patent Application No. 2021282413 (3 pages). [cited by applicant]
Communication Pursuant to Article 94(3) EPC mailed Jan. 31, 2023, issued in connection with European Patent Application No. 16873975.3 (8 pages). [cited by applicant]
Notice of Allowance mailed Feb. 14, 2023, issued in connection with U.S. Appl. No. 17/705,130 (5 pages). [cited by applicant]
Notice of Allowance mailed Mar. 22, 2023, issued in connection with U.S. Appl. No. 16/545,607 (9 pages). [cited by applicant]
Communication Pursuant to Article 94(3) EPC issued by the European Patent Office mailed on Apr. 28, 2023, issued in connection with European Patent Application No. 19866788.3 (5 pages). [cited by applicant]
Office Action mailed Jun. 30, 2023, issued in connection with U.S. Appl. No. 17/729,613 (49 pages). [cited by applicant]
Examination Report No. 1 mailed Apr. 12, 2023, issued by the Australian Patent Office in connection with Australian Patent Application No. 2018364813 (3 pages). [cited by applicant]
International Search Report of the International Searching Authority mailed on Mar. 27, 2017, issued in connection with International Application No. PCT/US2016/65947 (3 pages). [cited by applicant]
Written Opinion of the International Searching Authority mailed on Mar. 27, 2017, issued in connection with International Application No. PCT/US2016/65947 (7 pages). [cited by applicant]
Office Action mailed Sep. 26, 2018, issued in connection with U.S. Appl. No. 15/374,695 (33 pages). [cited by applicant]
Notice of Allowance mailed May 13, 2019, issued in connection with U.S. Appl. No. 15/374,695 (7 bages). [cited by applicant]
Extended European Search Report mailed Jun. 11, 2019, issued in connection with European Patent Application No. 16873975.3 (8 pages). [cited by applicant]
Communication Pursuant to Article 94(3) EPC issued by the European Patent Office mailed on Apr. 22, 2020, issued in connection with European Patent Application No. 16873975.3 (6 pages). [cited by applicant]
International Search Report of the International Searching Authority mailed on Dec. 12, 2019, issued in connection with International Application No. PCT/US2019/52670 (3 pages). [cited by applicant]
Written Opinion of the International Searching Authority mailed on Dec. 12, 2019, issued in connection with International Application No. PCT/US2019/52670 (5 pages). [cited by applicant]
Office Action mailed Feb. 5, 2020, issued in connection with U.S. Appl. No. 16/580,741 (15 pages). [cited by applicant]
International Search Report of the International Searching Authority mailed on May 14, 2015, issued in connection with International Application No. PCT/US15/015491(3 pages). [cited by applicant]
Written Opinion of the International Searching Authority mailed on May 14, 2015, issued in connection with International Application No. PCT/US15/015491 (9 pages). [cited by applicant]
Fung, et al., “A Mobile Assisted Localization Scheme for Augmented Reality,” Department of Computer Science and Engineering, The Chinese University of Hong Kong, 2012 (76 pages). [cited by applicant]
Sankar, et al., “Capturing Indoor Scenes With Smartphones,” UIST'12, Oct. 7-10, 2012, Cambridge, Massachusetts (9 pages). [cited by applicant]
Office Action mailed Aug. 8, 2017, issued in connection with U.S. Appl. No. 14/620,004 (26 pages). [cited by applicant]
Office Action mailed Aug. 28, 2018, issued in connection with U.S. Appl. No. 14/620,004 (33 pages). [cited by applicant]
Farin, et al., “Floor-Plan Reconstruction from Panoramic Images,” Sep. 23-28, 2007, MM '07, ACM (4 pages). [cited by applicant]
Office Action mailed Mar. 29, 2019, issued in connection with U.S. Appl. No. 14/620,004 (22 pages). [cited by applicant]
Office Action mailed Dec. 10, 2019, issued in connection with U.S. Appl. No. 14/620,004 (27 pages). [cited by applicant]
Zhang, et al., “Wallk&Sketch: Create Floor Plans with an RGB-D Camera,” Sep. 5-8, 2012, UbiComp '12, ACM (10 pages). [cited by applicant]
Office Action mailed Jul. 8, 2020, issued in connection with U.S. Appl. No. 14/620,004 (27 pages). [cited by applicant]
Office Action mailed Sep. 22, 2020, issued in connection with U.S. Appl. No. 16/580,741 (14 pages). [cited by applicant]
Office Action mailed Feb. 2, 2021, issued in connection with U.S. Appl. No. 14/620,004 (28 pages). [cited by applicant]
Communication Pursuant to Article 94(3) EPC issued by the European Patent Office mailed on Feb. 18, 2021, issued in connection with European Patent Application No. 16873975.3 (5 pages). [cited by applicant]
Examination Report No. 1 mailed Mar. 30, 2021, issued by the Australian Patent Office in connection with Australian Patent Application No. 2016366537 (6 pages). [cited by applicant]
Office Action mailed Apr. 21, 2021, issued in connection with U.S. Appl. No. 16/580,741 (15 pages). [cited by applicant]
Notice of Allowance mailed Aug. 19, 2021, issued in connection with U.S. Appl. No. 14/620,004 (11 pages). [cited by applicant]
Examiner-Initiated Interview Summary dated Aug. 10, 2021, issued in connection with U.S. Appl. No. 14/620,004 (1 page). [cited by applicant]
International Search Report of the International Searching Authority mailed on Feb. 11, 2019, issued in connection with International Application No. PCT/US18/60762 (3 pages). [cited by applicant]
Written Opinion of the International Searching Authority mailed on Feb. 11, 2019, issued in connection with International Application No. PCT/US18/60762 (7 pages). [cited by applicant]
Office Action mailed Apr. 16, 2020, issued in connection with U.S. Appl. No. 16/189,512 (10 pages). [cited by applicant]
U.S. Appl. No. 62/512,989, filed May 31, 2017 entiitled, “Systems and Methods for Rapidly Developing Annotated Computer Models of Structures” (47 pages). [cited by applicant]
Office Action mailed Dec. 14, 2020, issued in connection with U.S. Appl. No. 16/189,512 (10 pages). [cited by applicant]
Extended European Search Report dated Jul. 1, 2021, issued by the European Patent Office in connection with European Application No. 18876121.7 (8 pages). [cited by applicant]
Office Action mailed Jul. 20, 2021, issued in connection with U.S. Appl. No. 16/189,512 (11 pages). [cited by applicant]
Office Action mailed Dec. 27, 2021, issued in connection with U.S. Appl. No. 16/580,741 (13 bages). [cited by applicant]
Notice of Allowance mailed Dec. 16, 2021, issued in connection with U.S. Appl. No. 14/620,004 (12 pages). [cited by applicant]
Invitation to Pay Additional Fees issued by the International Searching Authority mailed on Feb. 2, 2022, issued in connection with International Application No. PCT/US21/63469 (2 pages). [cited by applicant]
Extended European Search Report dated Feb. 18, 2022, issued in connection with European Patent Application No. 19866788.3 (9 pages). [cited by applicant]
Office Action dated Feb. 3, 2022, issued by the Canadian Intellectual Property Office in connection with Canadian Application No. 3,007,968 (5 pages). [cited by applicant]
Office Action dated Mar. 25, 2022, issued in connection with U.S. Appl. No. 16/545,607 (56 pages). [cited by applicant]
International Search Report of the International Searching Authority mailed on Apr. 8, 2022, issued in connection with International Application No. PCT/US21/63469 (5 pages). [cited by applicant]
Written Opinion of the International Searching Authority mailed on Apr. 8, 2022, issued in connection with International Application No. PCT/US21/63469 (6 pages). [cited by applicant]
Notice of Allowance mailed Apr. 8, 2022, issued in connection with U.S. Appl. No. 16/189,512 (8 pages). [cited by applicant]
Notice of Allowance mailed Jun. 21, 2022, issued in connection with U.S. Appl. No. 16/189,512 (7 pages). [cited by applicant]
Dino, et al., “Image-Based Construction of Building Energy Models Using Computer Vision,” Automation in Construction (2020) (15 pages). [cited by applicant]
Fathi, et al., “Automated As-Built 3D Reconstruction of Civil Infrastructure Using Computer Vision: Achievements, Opportunities, and Challenges,” Advanced Engineering Informatics (2015) (13 pages). [cited by applicant]
International Search Report of the International Searching Authority mailed on Jul. 25, 2022, issued in connection with International Application No. PCT/US22/22024 (3 pages). [cited by applicant]
Written Opinion of the International Searching Authority mailed on Jul. 25, 2022, issued in connection with International Application No. PCT/US22/22024 (5 pages). [cited by applicant]
Office Action dated Sep. 2, 2022, issued in connection with U.S. Appl. No. 16/580,741 (13 pages). [cited by applicant]
Notice of Allowance mailed Sep. 6, 2022, issued in connection with U.S. Appl. No. 16/189,512 (7 pages). [cited by applicant]
Office Action mailed Dec. 7, 2023, issued by the Canadian Patent Office in connection with Canadian Patent Application No. 3,082,516 (6 pages). [cited by applicant]
Office Action dated Jan. 12, 2024, issued in connection with U.S. Appl. No. 18/214,842 (4 pages). [cited by applicant]
International Search Report of the International Searching Authority mailed on Dec. 15, 2023, issued in connection with International Application No. PCT/US2023/031123 (6 pages). [cited by applicant]
Written Opinion of the International Searching Authority mailed on Dec. 15, 2023, issued in connection with International Application No. PCT/US2023/031123 (7 pages). [cited by applicant]
Office Action dated Jan. 30, 2024, issued in connection with U.S. Appl. No. 17/895,500 (16 pages). [cited by applicant]
Office Action dated Mar. 12, 2024, issued in connection with U.S. Appl. No. 17/122,080 (10 pages). [cited by applicant]
Sampaio, et al., “Maintenance of Building Components Supported in Interactive Environments,” 10th Iberian Conference on Information Systems and Technologies, Jun. 17, 2015 (7 pages). [cited by applicant]
Notice of Allowance dated Mar. 18, 2024, issued in connection with U.S. Appl. No. 17/729,613 (10 pages). [cited by applicant]
Communication Pursuant to Article 94(3) EPC mailed Jan. 23, 2024, issued by the European Patent Office in connection with European Patent Application No. 19866788.3 (5 pages). [cited by applicant]
Office Action dated Jun. 4, 2024, issued by the Canadian Intellectual Property Office in connection with Canadian Application No. 3,082,516 (4 pages). [cited by applicant]
Notice of Allowance dated Jul. 3, 2024, issued in connection with U.S. Appl. No. 17/895,500 (6 pages). [cited by applicant]
Notice of Allowance dated Jul. 12, 2024, issued in connection with U.S. Appl. No. 17/729,613 (9 pages). [cited by applicant]
Examination Report No. 1 mailed May 20, 2025, issued by the Australian Patent Office in connection with Australian Patent Application No. 2024201803 (3 pages). [cited by applicant]
Communication Pursuant to Article 94(3) EPC mailed Jun. 6, 2025, issued by the European Patent Office in connection with European Patent Application No. 16873975.3 (9 pages). [cited by applicant]