IP Library › Granted Patent US 12,380,652
Granted Patent B2
US 12,380,652 · App. 17/987,256 · Granted Aug 5, 2025

Floorplan generation based on room scanning

Inventors: Feng Tang (Cupertino, CA); Afshin Dehghan (Sunnyvale, CA); Kai Kang (San Jose, CA); Yang Yang (Sunnyvale, CA); Yikang Liao (Sunnyvale, CA); Guangyu Zhao (Bejing, CN)
Assignee: Apple Inc.
G06T19/00G06F18/21G06F18/2148G06F18/217G06F18/24G06N3/045G06T7/50G06T7/73G06T11/00G06T15/205G06T19/003G06T19/20G06V10/22G06V20/10G06V20/36G06V20/64G06V30/274H04N7/183G06T2207/20081G06T2207/20084G06T2210/04G06T2210/12G06T2210/56
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,380,652
App. No.
17/987,256
Granted
Aug 5, 2025
Kind
B2
Abstract

Various implementations disclosed herein include devices, systems, and methods that generate floorplans and measurements using a three-dimensional (3D) representation of a physical environment generated based on sensor data.

Claims (70)

1. A method comprising:

at an electronic device having a processor:

displaying a live camera feed comprising a sequence of images of a physical environment;

obtaining a three-dimensional (3D) representation of the physical environment that was generated based on depth data and light intensity image data of the physical environment obtained during the displaying of the live camera feed;

generating a live preview of a first floorplan of the physical environment based on the 3D representation of the physical environment utilizing a first process, wherein the first process for generating the first floorplan comprises generating an edge map by:

identifying walls in the physical environment based on the 3D representation;

identifying wall attributes in the physical environment based on the 3D representation;

identifying objects in the physical environment based on the 3D representation; and

generating the live preview of the first floorplan based on the edge map that includes the identified walls, the identified wall attributes, and the identified objects;

displaying the live preview of the first floorplan concurrently with the live camera feed; and

generating a second floorplan of the physical environment based on the 3D representation of the physical environment utilizing a second process that is different than the first process, wherein the second process comprises:

refining the identified walls and the identified wall attributes;

identifying the objects in the physical environment based on the 3D representation and generating 3D object representations corresponding to the identified objects; and

generating the second floorplan based on the refined identified walls, the refined identified wall attributes, and the 3D object representations associated with the identified objects.

2. The method of claim 1 , wherein generating the second floorplan comprises:

generating semantic data for multiple horizontal layers of the physical environment based on the 3D representation; and

generating the second floorplan using the semantic data.

3. The method of claim 1 , wherein the 3D representation is associated with 3D semantic data that includes a 3D point cloud that includes semantic labels associated with at least a portion of 3D points within the 3D point cloud.

4. The method of claim 3 , wherein the semantic labels identify walls, wall attributes, objects, and classifications of the objects of the physical environment.

5. The method of claim 1 , wherein the second floorplan comprises a 3D floorplan.

6. The method of claim 1 , further comprising:

displaying a preview of the second floorplan concurrently with the live camera feed.

7. The method of claim 1 , wherein the second process further includes:

classifying corners and small walls based on the 3D representation using a more computationally intensive neural network;

updating the second floorplan based on the classified corners and small walls;

determining refinements for the second floorplan using a standardization algorithm; and

updating the second floorplan of the physical environment based on the determined refinements for the second floorplan.

8. The method of claim 1 , wherein generating the edge map by identifying the walls further comprises:

determining parametrically refined lines for the edge map using a line fitting algorithm; and

updating the edge map based on the parametrically refined lines.

9. The method of claim 1 , wherein updating the edge map by identifying the wall attributes comprises:

determining boundaries for the identified wall attributes using a wall attribute neural network and the sequence of images of the live camera feed; and

generating refined boundaries using a polygon heuristics algorithm based on the 3D representation associated with the identified wall attributes.

10. The method of claim 1 , wherein updating the edge map based on the identified objects comprises generating 2D representations of the 3D object representations.

11. A device comprising:

a non-transitory computer-readable storage medium; and

one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising:

displaying a live camera feed comprising a sequence of images of a physical environment;

obtaining a three-dimensional (3D) representation of the physical environment that was generated based on depth data and light intensity image data of the physical environment obtained during the displaying of the live camera feed;

generating a live preview of a first floorplan of the physical environment based on the 3D representation of the physical environment utilizing a first process, wherein the first process for generating the first floorplan comprises generating an edge map by:

identifying walls in the physical environment based on the 3D representation;

identifying wall attributes in the physical environment based on the 3D representation;

identifying objects in the physical environment based on the 3D representation; and

generating the live preview of the first floorplan based on the edge map that includes the identified walls, the identified wall attributes, and the identified objects;

displaying the live preview of the first floorplan concurrently with the live camera feed; and

generating a second floorplan of the physical environment based on the 3D representation of the physical environment utilizing a second process that is different than the first process, wherein the second process comprises:

refining the identified walls and the identified wall attributes;

identifying objects in the physical environment based on the 3D representation and generating 3D object representations corresponding to the identified objects; and

generating the second floorplan based on the refined identified walls, the refined identified wall attributes, and the 3D object representations associated with the identified objects.

12. The device of claim 11 , wherein generating the second floorplan comprises:

generating semantic data for multiple horizontal layers of the physical environment based on the 3D representation; and

generating the second floorplan using the semantic data.

13. The device of claim 11 , wherein the 3D representation is associated with 3D semantic data that includes a 3D point cloud that includes semantic labels associated with at least a portion of 3D points within the 3D point cloud.

14. The device of claim 13 , wherein the semantic labels identify walls, wall attributes, objects, and classifications of the objects of the physical environment.

15. The device of claim 11 , wherein the second floorplan comprises a 3D floorplan.

16. The device of claim 11 , wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, further cause the one or more processors to perform operations comprising:

displaying a preview of the second floorplan concurrently with the live camera feed.

17. A non-transitory computer-readable storage medium, storing program instructions executable on a device to perform operations comprising:

displaying a live camera feed comprising a sequence of images of a physical environment;

obtaining a three-dimensional (3D) representation of the physical environment that was generated based on depth data and light intensity image data of the physical environment obtained during the displaying of the live camera feed;

generating a live preview of a first floorplan of the physical environment based on the 3D representation of the physical environment utilizing a first process, wherein the first process for generating the first floorplan comprises generating an edge map by:

identifying walls in the physical environment based on the 3D representation;

identifying wall attributes in the physical environment based on the 3D representation;

identifying objects in the physical environment based on the 3D representation; and

generating the live preview of the first floorplan based on the edge map that includes the identified walls, the identified wall attributes, and the identified objects;

displaying the live preview of the first floorplan concurrently with the live camera feed; and

generating a second floorplan of the physical environment based on the 3D representation of the physical environment utilizing a second process that is different than the first process, wherein the second process comprises:

refining the identified walls and the identified wall attributes;

identifying objects in the physical environment based on the 3D representation and generating 3D object representations corresponding to the identified objects; and

generating the second floorplan based on the refined identified walls, the refined identified wall attributes, and the 3D object representations associated with the identified objects.

Continuity (3)
Continuation 17146582 · Jan 12, 2021
Provisional Application 62962485 · Jan 17, 2020
Related Publication 20230075601A1 · Mar 9, 2023
References Cited (85)
US 8705893B1 · Zhang · 2014 [cited by examiner]
US 9792501B1 · Maheriya et al. · 2017 [cited by applicant]
US 9866815B2 · Vrcelj et al. · 2018 [cited by applicant]
US 10192115B1 · Sheffield et al. · 2019 [cited by applicant]
US 10373380B2 · Kutliroff et al. · 2019 [cited by applicant]
US 10387582B2 · Lewis et al. · 2019 [cited by applicant]
US 10430641B2 · Gao · 2019 [cited by applicant]
US 10460180B2 · Gao · 2019 [cited by applicant]
US 10572970B2 · Sturm et al. · 2020 [cited by applicant]
US 10586351B1 · Brailovskiy · 2020 [cited by applicant]
US 10972638B1 · Waschura · 2021 [cited by applicant]
US 10981272B1 · Nagarajan · 2021 [cited by applicant]
US 20060077253A1 · VanRiper et al. · 2006 [cited by applicant]
US 20140043436A1 · Bell · 2014 [cited by examiner]
US 20140267228A1 · Ofek · 2014 [cited by applicant]
US 20140301633A1 · Furukawa · 2014 [cited by applicant]
US 20160055268A1 · Bell et al. · 2016 [cited by applicant]
US 20160092608A1 · Yamamoto · 2016 [cited by applicant]
US 20160343140A1 · Ciprari · 2016 [cited by applicant]
US 20170316115A1 · Lewis · 2017 [cited by applicant]
US 20180260613A1 · Gao · 2018 [cited by applicant]
US 20180315162A1 · Sturm · 2018 [cited by examiner]
US 20180330184A1 · Sturm · 2018 [cited by applicant]
US 20180336683A1 · Feng · 2018 [cited by applicant]
US 20180336724A1 · Spring et al. · 2018 [cited by applicant]
US 20190026958A1 · Gausebeck · 2019 [cited by applicant]
US 20190038181A1 · Domeika · 2019 [cited by applicant]
US 20190043259A1 · Wang et al. · 2019 [cited by applicant]
US 20190130233A1 · Stenger · 2019 [cited by applicant]
US 20190149745A1 · Green · 2019 [cited by applicant]
US 20190155302A1 · Lukierski · 2019 [cited by applicant]
US 20190164346A1 · Kim · 2019 [cited by applicant]
US 20190243928A1 · Rejeb Sfar · 2019 [cited by applicant]
US 20190340433A1 · Frank · 2019 [cited by applicant]
US 20200074668A1 · Stenger · 2020 [cited by applicant]
US 20200074707A1 · Lee · 2020 [cited by applicant]
US 20200160487A1 · Kanzawa · 2020 [cited by applicant]
US 20200174132A1 · Nezhadarya · 2020 [cited by applicant]
US 20200184651A1 · Mukasa · 2020 [cited by applicant]
US 20200211284A1 · Lin · 2020 [cited by applicant]
US 20200349351A1 · Brook · 2020 [cited by applicant]
US 20210019954A1 · Trinh · 2021 [cited by applicant]
US 20210199809A1 · Wynn · 2021 [cited by applicant]
CN 110060255A · 2019 [cited by applicant]
Sim, Daniel, “Real-Time 3D Reconstruction and Semantic Segmentation Using Multi-Layer Heightmaps”, Master Thesis; Sep. 2, 2015; pp. 1-73. [cited by applicant]
Liu, Chen; WU, Jiaye, and Furukawa, Yasutaka, “FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scans”, EECV 2018; https://link.springer.com/conference/eccv, pp. 1-17. [cited by applicant]
Indovina, I., “Generating accurate floor plans from 3D scans,” retrieved from the internet on Jun. 2, 2021: https://www.digitalbridge.com/blog/generating-accurate-floor-plans-from-3d-scans, 6 pages, Nov. 1, 2019. [cited by applicant]
Stojanovic, V. et al., “Generation of Approximate 2D and 3D Floor Plans from 3D Point Clouds,” Proceedings of the 14th International Joint Conference on Computer Graphics Theory and Applications, pp. 177-184, Jan. 1, 20… [cited by applicant]
Xu, Y. et al.. “Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints,” 2019 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, pp. 2811-2820, Oct. 2019. [cited by applicant]
European Patent Office, Partial Search Report and Provisional Opinion, European Patent Application No. 21151661.2, 17 pages, Aug. 3, 2021. [cited by applicant]
European Patent Office, Extended European Search Report and Search Opinion, European Patent Application No. 21151661.2, 16 pages, Oct. 28, 2021. [cited by applicant]
Hazirbas, C. et al., “FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-based CNN Architecture,” Advances in Databases and Information Systems, Springer International Publishing, 15 pages, 2017. [cited by applicant]
Song, S. and Xiao, J., “Sliding Shapes for 3D Object Detection in Depth Images,” ICIAP: International Conference on Image Analysis and Processing, 17th International Conference, Naples, Italy, 18 pages, Sep. 9, 2013-Sep… [cited by applicant]
European Patent Office, Examination Report (Communication pursuant to Article 94(3) EPC), European Patent Application No. 21151661.2, 5 pages, Jul. 13, 2022. [cited by applicant]
United States Patent and Trademark Office Non-Final Office Action pertaining to U.S. Appl. No. 17/146,604 issued Nov. 28, 2022; 23 pages. [cited by applicant]
United States Patent and Trademark Office Non-Final Office Action, U.S. Appl. No. 17/146,582, 17 pages, Apr. 4, 2022. [cited by applicant]
United States Patent and Trademark Office Notice of Allowance, U.S. Appl. No. 17/146,582, 9 pages, Aug. 17, 2022. [cited by applicant]
United States Patent and Trademark Office, Non-Final Office Action, U.S. Appl. No. 17/146,604, 11 pages, Nov. 28, 2022. [cited by applicant]
United States Patent and Trademark Office, Non-Final Office Action, U.S. Appl. No. 17/148,682, 14 p. Oct. 27, 2022. [cited by applicant]
U.S. Patent and Trademark Office, Notice of Allowance, U.S. Patent Application No. 17/150, 178, 10 p. Sep. 8, 2022. [cited by applicant]
Babacan, K et al., “Semantic Segmentation of Indoor Point Clouds Using Convolutional Neural Network,” ISPRS Annals of the Protogrammetry, Remote Sensing and Spatial Information Sciences, vol. IV-4/W4, 8 p. 2014. [cited by applicant]
Bylow, E et al., “Combining Depth Fusion and Photometric Stereo for Fine-Detailed 3D Models,” Scandinavian Conference on Image Analysis, Springer Nature Switzerland AG, pp. 261-274, 2019. [cited by applicant]
Hou, J et al., ,“Ed-Sis: 3D Semantic Instance Segmentation of RGB-D Scans,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitions, 10 p. 2019. [cited by applicant]
Son, H. and Kim, C., “Semantic as-built 3D modeling of structural elements of building based on local concavity and convexity,” Advanced Engineering Informatics, 34, pp. 114-124, 2017. [cited by applicant]
Turner, E. and Zakhor, A., “Floor Plan Generation and Room Labeling of Indoor Environments from Laser Ranger Data,” International Conference of Computer Graphics Theory and Application (GRAPP), IEEE, pp. 1-12, 2014. [cited by applicant]
Wang, L. et al., “Multi-View Fusion-Based 3D Object Detection for Robot Indoor Scene Perception,” Sensors, 19, p. 4092, 20 pages, 2019. [cited by applicant]
Xiong, X. et al., “Automatic creation of semantically rich 3D building models from laser scanner data,” Automation in Constructions, 31, pp. 325-337, 2013. [cited by applicant]
U.S. Patent and Trademark Office; Final Office Action issued Apr. 17, 2023 which pertains to U.S. Appl. No. 17/146,604, filed Jan. 12, 2021; 22 pgs. [cited by applicant]
U.S. Patent and Trademark Office; Final Office Action issued Mar. 16, 2023 which pertains to U.S. Appl. No. 17/148,682. 6 Pages. [cited by applicant]
U.S. Patent and Trademark Office, Advisory Action, U.S. Appl. No. 17/146,604, 3 pages, Jun. 27, 2023. [cited by applicant]
Ochmann, S. et al., “Automatic Generation of Structural Building Descriptions from 3D Point Cloud Scans,” Presented at the 2014 International Conference on Computer Graphics Theory and Applications (GRAPP), [retrieved f… [cited by applicant]
European Patent Office, EP Extended Search Report issued Aug. 9, 2023 which pertains to European Patent Application No. 23178270.7. 13 pages. [cited by applicant]
Xu, et al., Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints, 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV); pp. 2811-2820. [cited by applicant]
Ignazio, Generating accurate floor plans from 3D scans, Nov. 2019, https://www.digitalbridge.com/blog/generating-accurate-floor-plans-from-3d-scans, 6 pages. [cited by applicant]
Stojanovic, et al., Generation of Approximate 2D and 3D Floor Plans from 3D Point Clouds, Jan. 2019, Proceedings of the 14 [cited by applicant]
Liu, et al., FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scans, Oct. 2018, International Conference on Image Analysis and Processing, 17 [cited by applicant]
Singh, et al., Locating 3D Object Proposals: A Depth-Based Online Approach, Sep. 2017, IEEE Trans. on Circuits and System for Video Technology, pp. 1-14. [cited by applicant]
Deng, et al., Amodal Detection of 3D Objects: Inferring 3D Bounding Boxes from 2D Ones in RGB-Depth Images, Jul. 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 398-406. [cited by applicant]
Hazirbas, et al., FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-based CNN Architecture, Jan. 2017, Springer International Publishing, pp. 1-15. [cited by applicant]
Song, et al., Sliding Shapes for 3D Object Detection in Depth Images, International Conference on Image Analysis and Processing, 17 [cited by applicant]
China National Intellectual Property Administration, Patent Search Report (with English translation), Chinese Application No. 2021100576020, 6 pages, Jun. 25, 2024. [cited by applicant]
Rahman, M.M. et al., 3D object detection: Learning 3D bounding boxes from scaled down 2D bounding boxes in RGB-D images, Information Sciences, 476, pp. 147-158, 2019. [cited by applicant]
Yang, B. et al., Learning object bounding boxes for 3d instance segmentation on point clouds, Advances in neural information processing systems, 32, 10 pages, 2019. [cited by applicant]
Bassier, M. et al., Clustering of wall geometry from unstructured point clouds using conditional random fields, Remote Sensing, 11(13), p. 1586 (18 pages), 2019. [cited by applicant]
U.S. Patent and Trademark Office, Final Office Action, U.S. Appl. No. 18/422,429, 12 pages, Mar. 28, 2025. [cited by applicant]