IP Library Granted Patent US 9,171,405
Granted Patent B1
US 9,171,405 · App. 14/488,541 · Granted Oct 27, 2015

Identifying and filling holes across multiple aligned three-dimensional scenes

Inventors: Matthew Bell (Palo Alto, CA); David Gausebeck (Mountain View, CA)
Assignee: Matterport, Inc.
G06T19/20G01S17/89
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,171,405
App. No.
14/488,541
Granted
Oct 27, 2015
Kind
B1
Abstract

The capture and alignment of multiple 3D scenes is disclosed. Three dimensional capture device data from different locations is received thereby allowing for different perspectives of 3D scenes. An algorithm uses the data to determine potential alignments between different 3D scenes via coordinate transformations. Potential alignments are evaluated for quality and subsequently aligned subject to the existence of sufficiently high relative or absolute quality. A global alignment of all or most of the input 3D scenes into a single coordinate frame may be achieved. The presentation of areas around a particular hole or holes takes place thereby allowing the user to capture the requisite 3D scene containing areas within the hole or holes as well as part of the surrounding area using, for example, the 3D capture device. The new 3D captured scene is aligned with existing 3D scenes and/or 3D composite scenes.

Claims (55)

1. A method, comprising:

receiving, by a system comprising a processor, two or more sets of three dimensional data comprising points in a three dimensional coordinate space, wherein the points are respectively associated with three dimensional feature information, and wherein the two or more sets are associated with different capture positions relative to the three dimensional coordinate space;

determining, by the system, an alignment between the two or more sets of three dimensional data based on the three dimensional feature information for the points respectively included therein;

generating, by the system, a three dimensional representation of an environment with the two or more sets of three dimensional data based on the alignment; and

filling, by the system, a hole identified in the three dimensional representation of the environment.

2. The method of claim 1 , wherein the determining the alignment comprises:

evaluating, by the system, different alignments between the two or more sets of three dimensional data based on comparison of the feature information for the points respectively included therein as aligned with respect to each of the different alignments; and

selecting, by the system, the alignment from different alignments based on the evaluating.

3. The method of claim 1 , wherein the hole is formed via absence of three dimensional data in a portion of the three dimensional representation of the environment.

4. The method of claim 1 , wherein the hole is associated with three dimensional data of a particular quality.

5. The method of claim 1 , further comprising:

generating, by the system, a three dimensional visualization of an area surrounding the hole in the three dimensional representation of the environment; and

displaying, by the system, the three dimensional visualization on a graphics display device prior to filling the hole.

6. The method of claim 5 , further comprising prompting a user to capture new three dimensional data to fill the hole.

7. The method of claim 6 , further comprising:

receiving the new three dimensional data; and

merging the new three dimensional data with the two or more sets of three dimensional data to fill the hole.

8. The method of claim 6 , further comprising providing feedback as to when the hole has been filled.

9. The method of claim 1 , wherein the filling the hole comprises filling the hole using texture and geometry information from three dimensional data surrounding the hole included in the three dimensional representation of the environment.

10. The method of claim 1 , wherein the filling the hole comprises:

adjusting, using a robotic apparatus, a position of a capture device employed to capture the two or more sets of three dimensional data;

receiving new three dimensional data captured via the capture device following the reposition thereof; and

merging the new three dimensional data with the two or more sets of three dimensional data to fill the hole.

11. The method of claim 10 , wherein the adjusting the position of the capture device comprises determining a position and orientation for the three dimensional capture device based on a geometric property of the hole or an area surrounding the hole.

12. A non-transitory computer readable storage medium comprising computer-readable instructions that, in response to execution, cause a computing system to perform operations, comprising:

receiving three dimensional data captures of an environment captured from different positions relative to the environment, the three dimensional data captures respectively comprising points associated with three dimensional feature information;

aligning the three dimensional data captures relative to a three dimensional coordinate space, based on the three dimensional feature information for the points respectively associated therewith, to create a three dimensional representation of the environment; and

filling a hole identified in the three dimensional representation of the environment.

13. The non-transitory computer readable storage medium of claim 12 , wherein the aligning comprises:

generating different alignment schemes between the three dimensional data captures;

selecting one of the different alignment schemes based on comparison of the feature information for the points respectively associated with the three dimensional data captures as aligned with respect to each of the different alignments; and

aligning the three dimensional data captures based on the one of alignment schemes.

14. The non-transitory computer readable storage medium of claim 12 , wherein the hole is formed via absence three dimensional data corresponding to a portion of the environment included in the three dimensional representation of the environment.

15. The non-transitory computer readable storage medium of claim 12 , wherein the hole includes a portion of the three dimensional representation of the environment associated with three dimensional data of poor quality.

16. The non-transitory computer readable storage medium of claim 12 , wherein the operations further comprise:

displaying the hole to a user prior to filling the hole; and

prompting the user to capture new three dimensional data to fill the hole.

17. The non-transitory computer readable storage medium of claim 16 , wherein the operations further comprise:

receiving new three dimensional data captures of the environment in response to the prompting; and

merging the new three dimensional data captures with the three dimensional data captures to fill the hole.

18. The non-transitory computer readable storage medium of claim 12 , wherein the filling the hole comprises filling the hole using texture and geometry information from an area of the three dimensional representation of the environment surrounding the hole.

19. The non-transitory computer readable storage medium of claim 12 , wherein the filling the hole comprises:

adjusting, using a robotic apparatus, a position of a capture device employed to capture the three dimensional data captures;

receiving new three dimensional data captures captured via the capture device following the reposition thereof; and

merging the new three dimensional data captures with the three dimensional data captures to fill the hole.

20. The non-transitory computer readable storage medium of claim 19 , wherein the operations further comprise:

determining at least one geometric property of the hole or an area surrounding the hole, and wherein the directing the robotic apparatus comprises directing the robotic apparatus to place the capture device at a position and an orientation that is determined based on the at least one geometric property.

21. A system, comprising:

a memory to store executable instructions; and

a processor, coupled to the memory, that facilitates execution of the executable instructions to perform operations, comprising:

receiving sets of three dimensional data respectively comprising points associated with three dimensional feature information, wherein the sets of three dimensional data are associated with different capture positions relative to a three dimensional coordinate space;

determining an alignment between the sets of three dimensional data relative to the three dimensional coordinate space based on the three dimensional feature information for the points respectively included therein;

generating a three dimensional visualization with the sets of three dimensional data based on the alignment;

identifying a hole in the three dimensional visualization where three dimensional data is absent or of poor quality; and

filling the hole.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 25, 2025
From: MATTERPORT, LLC
To: COSTAR REALTY INFORMATION, INC.
Reel/Frame 072938/0375 →
MERGER AND CHANGE OF NAME Recorded Sep 10, 2025
From: MATTERPORT, INC.; MATRIX MERGER SUB II LLC
To: MATTERPORT, LLC
Reel/Frame 072827/0337 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 17, 2014
From: BELL, MATTHEW; GAUSEBECK, DAVID
To: MATTERPORT, INC.
Reel/Frame 033756/0554 →
Continuity (3)
Continuation 14053045 · Oct 14, 2013
Division 13539252 · Jun 29, 2012
Provisional Application 61502427 · Jun 29, 2011