IP Library › Granted Patent US 12,412,004
Granted Patent B2
US 12,412,004 · App. 18/501,738 · Granted Sep 9, 2025

Method for tracking construction site progress

Inventor: Anders Rong (Greve, DK)
Assignee: DALUX ApS
G06F30/13G06Q50/08G06T7/0002G06T17/00G06V20/647G06T2200/24G06T2207/20081G06T2207/30184G06T2210/04G06T2210/62
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,412,004
App. No.
18/501,738
Granted
Sep 9, 2025
Kind
B2
Abstract

Methods for tracking construction site progress, such as for a site that includes a building based on 3D digital representation. Building information modeling (BIM) may provide a digital representation of the physical and functional characteristics of a place, such as a site with a building, the building optionally based on 3D digital representation.

Claims (46)

1. A method for tracking construction site progress comprising:

a. providing a digital representation of a 3D model, optionally comprising metadata;

b. capturing one or more images comprising pixels at the site with a camera;

c. providing a location and a direction of said image;

d. rendering the 3D model into a virtual projection of a part comprising virtual pixels; wherein said virtual projection comprises for each virtual pixel:

i. coordinates of the rendered virtual projection of the 3D model;

ii. a surface normal of the rendered virtual projection of the 3D model; and

iii. any Building Information Modeling (BIM) identification marker of the rendered virtual projection of the 3D model;

e. feeding said rendered virtual projection into a digital processor, thereby allowing displaying or comparing a relevant part of the 3D model and the image in the same format;

f. using a machine learning model to identify objects of the 3D model in said captured images, said machine learning model having been trained on objects of at least one 3D model using information about a visual appearance, coordinates, and surface normals of said objects in said at least one 3D model; and

g. comparing the identified objects against the 3D model to conclude which objects of the 3D model are missing at the site.

2. The method according to claim 1 , wherein said virtual projection further comprises for each virtual pixel a distance from the rendered part of the 3D Model to the location.

3. The method according to claim 1 , wherein said virtual projection further comprises for each virtual pixel binary information indicating if the rendered part of the 3D model faces the camera.

4. The method according to claim 1 , comprising visually presenting (i) the 3D model, (ii) the one or more images and (iii) any identified missing objects by at least one of the following:

I. visual representations of (i) the 3D model with the corresponding (ii) image side by side;

II. a generated composite image of (i) and (ii), wherein the transparency of (i) or (ii) may be toggled by a user;

III. the images (ii) with any missing objects (iii) of the 3D model;

IV. a 2D plan providing information about any missing objects; and/or

V. a summary report comprising information about any missing object.

5. The method according to claim 1 , wherein said virtual projection is an equirectangular projection.

6. The method according to claim 5 , wherein said equirectangular projection is a cube map with six sides.

7. The method according to claim 1 , wherein said digital processor is selected among a CPU and a GPU.

8. A system comprising a processor and memory, the system configured to perform steps a., d., e., f., and g., of claim 1 , and to receive input of image data of steps b. and c. of claim 1 .

9. A non-transitory computer-readable storage medium storing executable instructions that, when executed by a hardware processor, cause the hardware processor to perform steps a., d., e., f., and g., of claim 1 using image data from steps b. and c. of claim 1 .

10. The method according to claim 1 , wherein said training of said machine learning model is done using the 3D model of step a., and/or one or more different 3D models.

11. The method according to claim 1 , further comprising:

h. classifying any identified objects of the 3D model into at least one of groups:

i. superfluous objects;

ii. misplaced objects;

iii wrong objects; and

iv. wrongly installed objects.

12. The method according to claim 1 , further comprising identifying any deform, poorly or faulty constructed objects.

13. The method according to claim 1 , further comprising identifying features of the 3D model.

14. The method according to claim 1 , wherein images from said camera are collected and stored together with input from sensors detecting motion of said camera.

15. The method according to claim 1 , wherein said camera is carried and moved by a drone or a person.

16. The method according to claim 1 , comprising a step of generating output which is readable on a mobile device to show missing, superfluous, misplaced objects, wrong and/or wrongly installed objects of the 3D model.

17. The method according to claim 1 , comprising a step of generating output which is readable on a browser running on a computer to provide a report showing missing, superfluous, misplaced objects, wrong and/or wrongly installed objects of the 3D model.

18. The method according to claim 1 , comprising applying a filter to generate output showing missing, superfluous, partly finished and/or misplaced objects and/or poorly or faulty constructed objects of the 3D model selectable from one or more categories.

19. The method according to claim 18 , wherein the categories are selected among walls, windows, doors, ventilation tubes, ventilation vents, water tubes, electrical installations, and fire alert systems.

20. The method according to claim 1 , wherein a user applies a filter to the 3D model to provide a simplified model of the 3D model.

21. The method according to claim 1 , wherein tracking site progress comprises monitoring progress of construction of infrastructure.

22. The method according to claim 1 , wherein quantities of objects are summarized, allowing correlation with budgets, supply chain and/or detection of waste or theft at the site.

23. The method according to claim 13 , wherein said features are selected among openings, voids, and holes.

24. The method according to claim 16 , wherein said mobile device is selected among a mobile phone and a tablet computer.

25. The method according to claim 20 , wherein said simplified model is selected among an architectural model, a structural model, a mechanical model, an electrical model, a plumbing model, a ventilation model and a fire protection model.

26. The method according to claim 21 , wherein said infrastructure is selected among roads, tunnels, bridges, railways, stations, and electrical grids.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 1, 2023
From: RONG, ANDERS
To: DALUX APS
Reel/Frame 065730/0829 →
Continuity (1)
Related Publication 20250148141A1 · May 8, 2025
References Cited (62)
US 6489955B1 · Newhall, Jr. · 2002 [cited by examiner]
US 9070216B2 · Golparvar-Fard et al. · 2015 [cited by applicant]
US 11436812B2 · Fleischman et al. · 2022 [cited by applicant]
US 20130155058A1 · Golparvar-Fard et al. · 2013 [cited by applicant]
US 20180012125A1 · Ladha et al. · 2018 [cited by applicant]
US 20180082414A1 · Rozenberg et al. · 2018 [cited by applicant]
US 20190100928A1 · Raman et al. · 2019 [cited by applicant]
US 20200151833A1 · Bellaish et al. · 2020 [cited by applicant]
US 20220064935A1 · Williams et al. · 2022 [cited by applicant]
US 20220198709A1 · Sudry et al. · 2022 [cited by applicant]
US 20220375183A1 · Fleischman et al. · 2022 [cited by applicant]
US 20230394765A1 · Tang · 2023 [cited by examiner]
WO 2013041101A1 · 2013 [cited by applicant]
Karakotta et al.; “360 degree Surface Regression with a Hyper-Sphere Loss”; 2019 International Conference of 3d Vision (3DV), pp. 258-268. (Year: 2019). [cited by examiner]
Zhang et al.; “Advanced Progress Control of Infrastructure Construction Projects Using Terrestrial Laser Scanning Technology;” Oct. 12, 2020; MDPI infrastructures; pp. 1-18 (Year: 2020). [cited by examiner]
Kim et al. “Recognizing and Classifying Unknown Object in BIM Using 2D CNN”; May 21, 2019; In: Lee, JH. (eds) Computer-Aided Architectural Design. “Hello, Culture”. CAAD Futures 2019. Communications in Computer and Info… [cited by examiner]
Campos, C et al., “ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM” arXiv:2007.11898v2 [cs.RO] Apr. 23, 2021; DOI: 10.1109/TRO.2021.3075644; 2007.11898.pdf (arxiv.org), 18 pages. [cited by applicant]
Campos, C., et al. “Inertial-Only Optimization for Visual-Inertial Initialization”, arXiv:2003.05766v1 [cs.RO] Mar. 12, 2020, https://arxiv.org/pdf/2003.05766.pdf, 8 pages. [cited by applicant]
Chen1, Y. et al., “Bundle Adjustment Revisited” arXiv:1912.03858v1 [cs.CV] Dec. 9, 2019; 1912.03858.pdf (arxiv.org), 10 pages. [cited by applicant]
Dong, W. et al., “ASH: A Modern Framework for Parallel Spatial Hashing in 3D Perception”; arXiv:2110.00511v2 [cs.CV] Jan. 30, 2023; 2110.00511.pdf (arxiv.org), 18 pages. [cited by applicant]
Elfring, J. et al., “Particle Filters: A Hands-On Tutorial”, Sensors, 2021, 21, 438, https://doi.org/10.3390/s21020438, 28 pages. [cited by applicant]
Forster, C. et al., “On-Manifold Preintegration for Real-Time Visual-Inertial Odometry”; IEEE Transactions on Robotics, vol. 33, No. 1, Feb. 2017; TRO16_forster.pdf (uzh.ch), 20 pages. [cited by applicant]
Fraundorfer, F. and Scaramuzza, D., “Visual Odometry; Part II: Matching, Robustness, Optimization, and Applications” IEEE Robotics & Automation Magazine Jun. 2012, 13 pages. [cited by applicant]
Gálvez-López, D. and Tardós, JD., “Bags of Binary Words for Fast Place Recognition in Image Sequences”; IEEE Transactions on Robotics, vol. 28, Issue 5, Oct. 2012. Short Paper; main.dvi (doriangalvez.com), 9 pages. [cited by applicant]
Geneva, P et al., “IMU Noise Model”; IMU Noise Model ⋅ ethz-asl/kalibr Wiki ⋅ GitHub Accessed: Oct. 27, 2023, 6 pages. [cited by applicant]
Godsil, S. et al., “Maximum a Posteriori Sequence Estimation Using Monte Carlo Particle Filters”, Annals of the Institute of Statistical Mathematics 53, 82-96 (2001), 7 pages. [cited by applicant]
Hargreaves, S and Harris, M., “Deferred Shading” 6800 Leagues under the sea; Microsoft PowerPoint—deferred_shading.ppt (nvidia.com), 44 pages. [cited by applicant]
Hargreaves, S., “Deferred Shading”; Game Developers Conference 2014; Deferred Shading (GDC 2004) (utah.edu), 32 pages. [cited by applicant]
Hartley, R. and Zisserman, A., “Multiple View Geometry in Computer Vision” 2nd Edition (Book); 2004; DOI: https://doi.org/10.1017/CBO9780511811685. [cited by applicant]
He, K. et al., “Deep Residual Learning for Image Recognition”; arXiv: 1512.03385v1 [cs.CV] Dec. 10, 2015; 1512.03385.pdf (arxiv.org), 12 pages. [cited by applicant]
He, K. et al., “Mask R-CNN”; arXiv:1703.06870v3 [cs.CV] Jan. 24, 2018; 1703.06870.pdf (arxiv.org), 12 pages. [cited by applicant]
Ho, NH. et al., “Step-Detection and Adaptive Step-Length Estimation for Pedestrian Dead-Reckoning at Various Walking Speeds Using a Smartphone”; Sensors 2016, 16(9), 1423; https://doi.org/10.3390/s16091423, 14 pages. [cited by applicant]
Khedr, M. and El-Sheimy, N., “A Smartphone Step Counter Using IMU and Magnetometer for Navigation and Health Monitoring Applications”; Sensors 2017, 17(11), 2573; https://doi.org/10.3390/s17112573, 25 pages. [cited by applicant]
Khedr, M. and El-Sheimy, N., “S-PDR: SBAUPT-Based Pedestrian Dead Reckoning Algorithm for Free-Moving Handheld Devices”; Geomatics 2021, 1(2), 148-176; https://doi.org/10.3390/geomatics1020010, 30 pages. [cited by applicant]
Kirillov, A. et al., “Segment Anything” arXiv:2304.02643v1 [cs.CV] Apr. 5, 2023; arxiv.org/pdf/2304.02643.pdf, 30 pages. [cited by applicant]
Klajnscek, T., “Battle-Tested Deferred Rendering on PS3, Xbox 360 and PC”; GDC Europa 2009; GDC Vault—Battle-Tested Deferred Rendering on PS3, Xbox 360 and PC, 3 pages. [cited by applicant]
Kümmerle, R. et al., “g20: A General Framework for Graph Optimization” 2011 IEEE International Conference on Robotics and Automation, May 2011, Shanghai, China; kuemmerle11icra.pdf (uni-freiburg.de), 7 pages. [cited by applicant]
Lefebvre, S. et Hoppe, H., “Perfect Spatial Hashing”; ACM Trans. Graphics (SIGGRAPH), 25(3), 2006; Microsoft Research; Perfect Spatial Hashing (hhoppe.com), 10 pages. [cited by applicant]
Müller, M., “Blazing Fast Neighbor Search with Spatial Hashing”; Ten Minute Physics; PowerPoint Presentation (matthias-research.github.io) Accessed: Oct. 27, 2023, 11 pages. [cited by applicant]
Mur-Artal, R. and Tardós JD, “ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras”, arXiv:1610.06475v2 [cs.RO] Jun. 19, 2017; DOI: 10.1109/TRO.2017.2705103; arxiv.org/pdf/1610.06475.pdf, 9 page… [cited by applicant]
Mur-Artal, R. et al., “ORB-SLAM: a Versatile and Accurate Monocular SLAM System”, arXiv:1502.00956v2 [cs.RO] Sep. 18, 2015; DOI: 10.1109/TRO.2015.2463671; arxiv.org/pdf/1502.00956.pdf, 18 pages. [cited by applicant]
Redmon, J. et al., “You Only Look Once: Unified, Real-Time Object Detection”; arXiv:1506.02640v5 [cs.CV] May 9, 2016; 1506.02640.pdf (arxiv.org), 10 pages. [cited by applicant]
Ren, S. et al., “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”; arXiv:1506.01497v3 [cs.CV] Jan. 6, 2016; 1506.01497.pdf (arxiv.org), 14 pages. [cited by applicant]
Ronneberger, O. Et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation” arXiv:1505.04597v1 [cs.CV] May 18, 2015; https://arxiv.org/pdf/1505.04597.pdf, 8 pages. [cited by applicant]
Scaramuzza, D. and Fraundorfer, F., “Visual Odometry; Part I: The First 30 Years and Fundamentals” IEEE Robotics & Automation Magazine Dec. 2011, 13 pages. [cited by applicant]
Schönberger, JL. And Frahm, JM., “Structure-from-Motion Revisited” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Accessed: Oct. 27, 2023; Jun. 2016; DOI:10.1109/CVPR.2016.445;; schoenberger2016… [cited by applicant]
Stachniss, C., Mobile Robotics Block YouTube; Accessed: Oct. 27, 2023, retrieved at: https://www.youtube.com/watch?v=aGNPtwQyl-4&list=PLgnQpQtFTOGSeTU35ojkOdsscnenP2Cqx. [cited by applicant]
Stachniss, C., Mobile Sensing and Robotics 1 Course, YouTube; Accessed: Oct. 27, 2023, retrieved at: https://www.youtube.com/watch?v=OSsQX-dMwco&list=PLgnQpQtFTOGQJXx-x0t23RmRbjp_yMb4v. [cited by applicant]
Stachniss, C., Mobile Sensing and Robotics 2 Course, YouTube; Accessed: Oct. 27, 2023, retrieved at: https://www.youtube.com/watch?v=mQvKhmWagB4&list=PLgnQpQtFTOGQh_J16IMwDlji18SWQ2PZ6. [cited by applicant]
Stachniss, C., Occupancy Grid Maps, YouTube; Accessed: Oct. 27, 2023, retrieved at: https://www.youtube.com/watch?v=v-Rm9TUG9LA. [cited by applicant]
Stachniss, C., SLAM Course (2013), YouTube; Accessed: Oct. 27, 2023, retrieved at: https://www.youtube.com/playlist?list=PLgnQpQtFTOGQrZ405QzblHgl3b1JHimN_. [cited by applicant]
Sumikura, S. et al., “OpenVSLAM: A Versatile Visual SLAM Framework” arXiv:1910.01122v3 [cs.CV] Apr. 6, 2023; OpenVSLAM: A Versatile Visual SLAM Framework (arxiv.org). [cited by applicant]
Sun, C. et al., “HoHoNet: 360 Indoor Holistic Understanding with Latent Horizontal Features”; arXiv:2011.11498v3 [cs.CV] Sep. 9, 2021; 2011.11498.pdf (arxiv.org), 15 pages. [cited by applicant]
Tan, M. and Le, QV., “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”; arXiv:1905.11946v5 [cs.LG] Sep. 11, 2020; EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (arxiv.… [cited by applicant]
Thrun, S. et al., “Probabilistic Robotics” (Book); 2005, probabilistic-robotics.org. [cited by applicant]
Urban S. and Hinz S., “Multicol-Slam—A Modular Real-Time Multi-Camera Slam System” arXiv:1610.07336v1 [cs.CV] Oct. 24, 2016; 1610.07336.pdf (arxiv.org), 15 pages. [cited by applicant]
Valiant, M., “Deferred Rendering in Killzone 2”; Guerrilla, Develop Conference, Jul. 2007, Brighton; Develop07_Valient_DeferredRenderingInKillzone2.pdf (guerrilla-games.com), 55 pages. [cited by applicant]
Vezočnik, M. and Juric, MB., “Adaptive Inertial Sensor-Based Step Length Estimation Model” Sensors 2022, 22 (23), 9452; https://doi.org/10.3390/s22239452, 20 pages. [cited by applicant]
Wang, CY. Et al., “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors”; arXiv:2207.02696v1 [cs.CV] Jul. 6, 2022; 2207.02696.pdf (arxiv.org), 15 pages. [cited by applicant]
Wikipedia, “Deferred shading”; Deferred shading—Wikipedia; https://en.wikipedia.org/wiki/Deferred_shading, Retrieved on: Oct. 27, 2023, 8 pages. [cited by applicant]
Zhang, Y. and Huang, F., “Panoramic Visual SLAM Technology for Spherical Images” Sensors 2021, 21, 705. https://doi.org/10.3390/s21030705, 18 pages. [cited by applicant]
Zhao, Q. et al., “SPHORB: A Fast and Robust Binary Feature on the Sphere” International Journal of Computer Vision 113, 143-159 (2015), 6 pages. [cited by applicant]
Cited By (1)
US 12,636,778