IP Library › Granted Patent US 12,348,698
Granted Patent B2
US 12,348,698 · App. 18/308,639 · Granted Jul 1, 2025

Capturing and aligning panoramic image and depth data

Inventors: Kyle Simek (San Jose, CA); David Alan Gausebeck (Mountain View, CA); Matthew Tschudy Bell (Palo Alto, CA)
Assignee: Matterport, Inc.
H04N13/106H04N5/265H04N13/232H04N13/239H04N13/254H04N13/271H04N23/45G06T2207/20221H04N23/698
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,348,698
App. No.
18/308,639
Granted
Jul 1, 2025
Kind
B2
Abstract

This application generally relates to capturing and aligning panoramic image and depth data. In one embodiment, a device is provided that comprises a housing and a plurality of cameras configured to capture two-dimensional images, wherein the cameras are arranged at different positions on the housing and have different azimuth orientations relative to a center point such that the cameras have a collective field-of-view spanning up to 360° horizontally. The device further comprises a plurality of depth detection components configured to capture depth data, wherein the depth detection components are arranged at different positions on the housing and have different azimuth orientations relative to the center point such that the depth detection components have the collective field-of-view spanning up to 360° horizontally.

Claims (34)

1. A device having a vertical y axis extending through a center point, the device comprising:

a housing including:

at least one camera having a fisheye camera lens configured to capture 2D image data of an environment from a fixed location, the at least one camera positioned at a first fixed distance from the center point;

at least one depth sensor device including at least one light imaging detection and ranging (LiDAR) device configured to capture 3D depth data, the at least one depth sensor device positioned at a second fixed distance from the center point;

a horizontal rotatable mount configured to enable the fisheye camera lens of the at least one camera to rotate about a vertical y axis, the at least one camera being capable of capturing a plurality of images with mutually overlapping fields of view at different viewpoints; and

at least one processor configured to map, during capture of the 2D image data by the at least one camera and the 3D depth data by the at least one depth sensor device, the 2D image data from the at least one camera and the 3D depth data from the at least one depth sensor device to a common spatial 3D coordinate space based on known capture positions and orientations of the at least one camera and the at least one depth sensor device determined based on the first fixed distance and the second fixed distance to facilitate associating 3D coordinates with respective visual features included in the 2D image data relative to the common spatial 3D coordinate space.

2. The device of claim 1 , further including a vertical rotatable mount configured to enable the fisheye camera lens of the at least one camera to rotate about a horizontal x axis.

3. The device of claim 2 , wherein the horizontal rotatable mount and the vertical rotatable mount each include at least one motor configured to rotate the fisheye camera lens of the at least one camera.

4. The device of claim 1 , wherein the fisheye camera lens provides a field of a view from about 100 degrees to about 195 degrees.

5. The device of claim 1 , wherein the at least one processor is further configured to merge and align a plurality of 2D images captured by the at least one camera to generate a panoramic 2D image of the environment.

6. The device of claim 5 , wherein the at least one processor is further configured to merge and align the 3D depth data based on information from the at least one depth sensor device to generate a 3D image of the environment.

7. The device of claim 1 , wherein the at least one camera is a color video camera capable of capturing color video of the environment.

8. The device of claim 1 , wherein the at least one processor is further configured to determine each capture location relative to each other using positional tracking.

9. The device of claim 8 , wherein the positional tracking includes inertial position tracking.

10. The device of claim 8 , wherein the positional tracking includes utilizing simultaneous localization and mapping (SLAM) analysis.

11. A method performed by a device having a vertical y axis extending through a center point, the device comprising a housing, the housing including at least one camera, at least one depth sensor device, a horizontal rotatable mount, and at least one processor, the method comprising:

capturing, by the at least one camera, 2D image data of an environment from a fixed location, the at least one camera having a fisheye camera lens, the at least one camera positioned at a first fixed distance from the center point;

capturing, by the at least one depth sensor device, 3D depth data of the environment, the at least one depth sensor device including at least one light imaging detection and ranging (LiDAR) device, the at least one depth sensor device positioned at a second fixed distance from the center point;

rotating, by the horizontal rotatable mount, the fisheye camera lens of the at least one camera about a vertical y axis, the at least one camera being capable of capturing a plurality of images with mutually overlapping fields of view at different viewpoints; and

mapping, by the at least one processor and during capture of the 2D image data by the at least one camera and the 3D depth data by the at least one depth sensor device, the 2D image data from the at least one camera and the 3D depth data from the at least one depth sensor device to a common spatial 3D coordinate space based on known capture positions and orientations of the at least one camera and the at least one depth sensor device determined based on the first fixed distance and the second fixed distance to facilitate associating 3D coordinates with respective visual features included in the 2D image data relative to the common spatial 3D coordinate space.

12. The method of claim 11 , further comprising rotating, by a vertical rotatable mount, the fisheye camera lens of the at least one camera about a horizontal x axis.

13. The method of claim 12 , wherein the horizontal rotatable mount and the vertical rotatable mount each include at least one motor configured to rotate the fisheye camera lens of the at least one camera.

14. The method of claim 11 , wherein the fisheye camera lens provides a field of a view from about 100 degrees to about 195 degrees.

15. The method of claim 11 , further comprising merging and aligning, by the at least one processor, a plurality of 2D images captured by the at least one camera to generate a panoramic 2D image of the environment.

16. The method of claim 15 , further comprising merging and aligning, by the at least one processor, the 3D depth data based on information from the at least one depth sensor device to generate a 3D image of the environment.

17. The method of claim 11 , wherein the at least one camera is a color video camera capable of capturing color video of the environment.

18. The method of claim 11 , further comprising determining, by the at least one processor, each capture location relative to each other using positional tracking.

19. The method of claim 18 , wherein using the positional tracking includes inertial position tracking.

20. The method of claim 18 , wherein using the positional tracking includes utilizing simultaneous localization and mapping (SLAM) analysis.

21. A non-transitory computer-readable medium comprising instructions executable by one or more processors of a device having a vertical y axis extending through a center point to perform a method, the device comprising a housing, the housing including at least one camera, at least one depth sensor device, a horizontal rotatable mount, and at least one processor, the method, the method comprising:

capturing, by the at least one camera, 2D image data of an environment from a fixed location, the at least one camera having a fisheye camera lens, the at least one camera positioned at a first fixed distance from the center point;

capturing, by the at least one depth sensor device, 3D depth data of the environment, the at least one depth sensor device including at least one light imaging detection and ranging (LiDAR) device, the at least one depth sensor device positioned at a second fixed distance from the center point;

rotating, by the horizontal rotatable mount, at least the fisheye camera lens of the at least one camera about a vertical y axis, the at least one camera being capable of capturing a plurality of images with mutually overlapping fields of view at different viewpoints; and

mapping, by the at least one processor and during capture of the 2D image data by the at least one camera and the 3D depth data by the at least one depth sensor device, the 2D image data from the at least one camera and the 3D depth data from the at least one depth sensor device to a common spatial 3D coordinate space based on known capture positions and orientations of the at least one camera and the at least one depth sensor device determined based on the first fixed distance and the second fixed distance to facilitate associating 3D coordinates with respective visual features included in the 2D image data relative to the common spatial 3D coordinate space.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 25, 2025
From: MATTERPORT, LLC
To: COSTAR REALTY INFORMATION, INC.
Reel/Frame 072938/0425 →
MERGER AND CHANGE OF NAME Recorded Sep 10, 2025
From: MATTERPORT, INC.; MATRIX MERGER SUB II LLC
To: MATTERPORT, LLC
Reel/Frame 072827/0559 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 23, 2023
From: SIMEK, KYLE; GAUSEBECK, DAVID ALAN; BELL, MATTHEW TSCHUDY
To: MATTERPORT, INC.
Reel/Frame 063733/0893 →
Continuity (6)
Continuation 16559135 · Sep 3, 2019
Continuation 15417162 · Jan 26, 2017
Continuation 14070426 · Nov 1, 2013
Continuation 13776688 · Feb 25, 2013
Provisional Application 61603221 · Feb 24, 2012
Related Publication 20230269353A1 · Aug 24, 2023
References Cited (95)
US 6396496B1 · Pfister et al. · 2002 [cited by applicant]
US 7054478B2 · Harman · 2006 [cited by applicant]
US 7697749B2 · Ogawa · 2010 [cited by applicant]
US 8602887B2 · Tardif et al. · 2013 [cited by applicant]
US 8957919B2 · Han et al. · 2015 [cited by applicant]
US 8973029B2 · Deshpande et al. · 2015 [cited by applicant]
US 9001227B2 · Aleksic et al. · 2015 [cited by applicant]
US 9609307B1 · Lopez et al. · 2017 [cited by applicant]
US 10154244B2 · Peterson · 2018 [cited by applicant]
US 10860100B2 · Osterhout et al. · 2020 [cited by applicant]
US 20030227615A1 · Montgomery et al. · 2003 [cited by applicant]
US 20030231788A1 · Yukhin et al. · 2003 [cited by applicant]
US 20040023612A1 · Kriesel · 2004 [cited by applicant]
US 20050257748A1 · Kriesel et al. · 2005 [cited by applicant]
US 20060244746A1 · England et al. · 2006 [cited by applicant]
US 20080170123A1 · Albertson et al. · 2008 [cited by applicant]
US 20080310757A1 · Wolberg · 2008 [cited by examiner]
US 20100097444A1 · Lablans · 2010 [cited by applicant]
US 20100194860A1 · Mentz et al. · 2010 [cited by applicant]
US 20110018973A1 · Takayama · 2011 [cited by applicant]
US 20110102763A1 · Brown et al. · 2011 [cited by applicant]
US 20110157696A1 · Bennett et al. · 2011 [cited by applicant]
US 20110211042A1 · Thorpe et al. · 2011 [cited by applicant]
US 20120105574A1 · Baker · 2012 [cited by examiner]
US 20120162366A1 · Ninan et al. · 2012 [cited by applicant]
US 20120185094A1 · Rosenstein · 2012 [cited by examiner]
US 20120223937A1 · Bendall · 2012 [cited by applicant]
US 20130063550A1 · Ritchey et al. · 2013 [cited by applicant]
US 20130103303A1 · Lynch · 2013 [cited by examiner]
US 20150288877A1 · Glazer · 2015 [cited by applicant]
US 20160005179A1 · Petyushko et al. · 2016 [cited by applicant]
US 20160249039A1 · Tran et al. · 2016 [cited by applicant]
US 20160291136A1 · Lindskog et al. · 2016 [cited by applicant]
US 20170278289A1 · Marino et al. · 2017 [cited by applicant]
US 20180139431A1 · Simek et al. · 2018 [cited by applicant]
US 20180144555A1 · Ford et al. · 2018 [cited by applicant]
US 20180253869A1 · Yumer et al. · 2018 [cited by applicant]
US 20180276841A1 · Krishnaswamy et al. · 2018 [cited by applicant]
US 20190259170A1 · Qi · 2019 [cited by applicant]
WO 2007102663A1 · 2007 [cited by applicant]
Birchfield, Stan et al. Depth Discontinuities by Pixel-To-Pixel Stereo. International Journal of Computer Vision. Dec. 1999. vol. 35, Issue 3 pp. 269-293. [cited by applicant]
Davidson, Andrew et al. MonoSLAM: Real-Time Single Camera Slam. IEEE Transactions on Pattern Analysis and Machine Intelligence. Jun. 2007, vol. 29, IEEE Computer Society. [cited by applicant]
European Patent application No. 19864640.8, Extended European Search Report dated May 23, 2022, 13 Pages. [cited by applicant]
Final Office Action for U.S. Appl. No. 14/070,426 dated May 4, 2017, 78 pages. [cited by applicant]
Final Office Action for U.S. Appl. No. 14/070,426 dated Sep. 13, 2018, 52 pages. [cited by applicant]
Final Office Action for U.S. Appl. No. 14/070,426, dated Jun. 30, 2016, 38 pages. [cited by applicant]
Final Office Action for U.S. Appl. No. 14/070,427, dated Aug. 11, 2016, 40 pages. [cited by applicant]
Final Office Action for U.S. Appl. No. 14/070,427, dated May 18, 2017, 55 pages. [cited by applicant]
Final Office Action for U.S. Appl. No. 14/070,428, dated Sep. 5, 2017, 68 pages. [cited by applicant]
Final Office Action for U.S. Appl. No. 14/070,428, dated Sep. 9, 2016, 40 pages. [cited by applicant]
Final Office Action for U.S. Appl. No. 14/070,429, dated Aug. 31, 2017, 77 pages. [cited by applicant]
Final Office Action for U.S. Appl. No. 14/070,429, dated Sep. 8, 2016, 52 pages. [cited by applicant]
Final Office Action for U.S. Appl. No. 14/070,430, dated Dec. 28, 2017, 58 pages. [cited by applicant]
Final Office Action for U.S. Appl. No. 14/070,430, dated Oct. 20, 2016, 45 pages. [cited by applicant]
Final Office Action for U.S. Appl. No. 15/417,162 dated Dec. 4, 2019, 31 pages. [cited by applicant]
Final Office Action for U.S. Appl. No. 17/165,653 dated Jul. 26, 2023, 15 pages. [cited by applicant]
Hirschmuller, Heiko. Accurate and Efficient Stereo Processing By Semi-Global Matching and MutualInformation. IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2005. vol. 2. pp. 807-814. IEEE C… [cited by applicant]
Hsoni, Asmaa et al. Secrets of Adaptive Support Weight Techniques For Local Stereo Matching. Digital Representations of the Real World: How to Capture, Model, and Render Visual Reality. Computer Vision and Image Underst… [cited by applicant]
International Search Report and Written Opinion for International Patent Application Serial No. PCT/US2019/053040 dated Jan. 6, 2020, 7 pages. [cited by applicant]
International Serch Report and Written Opinion for International Patent Application Serial No. PCT/US2018/015330 dated Apr. 19, 2018, 11 pages. [cited by applicant]
Kolmogorov, V. et al. Computing Visual Correspondence with Occlusion Using Graph Cuts. 3D-TV System with Depth-Image-Based Rendering Architectures, Techniques and Challenges. Computer Vision. 2001. ICCV, IEEE. [cited by applicant]
Lin, Guo-Shiang et al., “2D to 3D Image Conversation Based on Classification of Background Depth Profiles,” Proceedings of the 5th Pacific Rim Symposium on Advances in Image and Video Technology (PSIVT 2011), Part II, p… [cited by applicant]
Non Final Office Action for U.S. Appl. No. 16/141,630, dated Aug. 26, 2021, 14 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 13/776,688, dated May 29, 2015, 102 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 14/070,426 dated Oct. 11, 2016, 45 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 14/070,426, dated Dec. 17, 2015, 38 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 14/070,426, dated Jan. 18, 2018, 50 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 14/070,427, dated Feb. 11, 2016, 42 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 14/070,427, dated Nov. 10, 2016, 43 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 14/070,428, dated Feb. 11, 2016, 44 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 14/070,428, dated Jan. 12, 2017, 46 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 14/070,429, dated Jan. 12, 2017, 55 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 14/070,429, dated Jan. 26, 2016, 53 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 14/070,430, dated Jan. 21, 2016, 17 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 14/070,430, dated Jun. 15, 2017, 48 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 15/417,162 dated May 9, 2019, 53 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 16/141,558 dated Sep. 25, 2020, 31 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 16/141,630 dated Jan. 26, 2021, 10 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 16/141,638, dated Feb. 5, 2021, 13 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 16/141,649 dated Mar. 4, 2021, 15 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 16/559, 135 dated Apr. 14, 2022, 14 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/165,653 dated Oct. 11, 2022, 32 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/165,653 dated Oct. 11, 2023, 32 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 13/776,688, dated Dec. 18, 2015, 28 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 14/070,426 dated Jul. 11, 2019, 24 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 14/070,427 dated Sep. 5, 2019, 20 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 14/070,428 dated Sep. 5, 2019, 19 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 14/070,429 dated Sep. 4, 2019, 20 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 14/070,430, dated Oct. 5, 2020, 10 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 15/417,162 dated Jul. 14, 2020, 9 pages. [cited by applicant]
Final Office Action for U.S. Appl. No. 17/654,989 dated Jan. 8, 2024, 9 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 17/165,653 dated Feb. 5, 2024, 8 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 17/654,989 dated Mar. 21, 2024, 8 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 18/669,442 dated Dec. 20, 2024, 8 pages. [cited by applicant]
European Patent Application No. 19864640.8, Examination Report dated May 16, 2024, 9 pages. [cited by applicant]