IP Library Granted Patent US 11,625,862
Granted Patent B2
US 11,625,862 · App. 17/072,439 · Granted Apr 11, 2023

Mirror reconstruction

Inventors: Michael Goesele (Darmstadt, DE); Julian Straub (Seattle, WA); Thomas John Whelan (Minane Bridge, IE); Richard Andrew Newcombe (Seattle, WA); Steven John Lovegrove (Woodinville, WA)
Assignee: Meta Platforms Technologies, LLC
G06T7/97G06T7/73G06T15/06G06T15/50G06T17/10
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,625,862
App. No.
17/072,439
Granted
Apr 11, 2023
Kind
B2
Abstract

In one embodiment, a method includes accessing a digital image captured by a camera that is connected to a machine-detectable object, detecting a reflection of the machine-detectable object in the digital image, computing, in response to the detection, a plane that is coincident with a reflective surface associated with the reflection, determining a boundary of the reflective surface in the plane based on at least one of a plurality of cues, and storing information associated with the reflective surface, where the information includes a pose of the reflective surface and the boundary of the reflective surface in a 3D model of a physical environment, and where the information associated with the reflective surface and the 3D model are configured to be used to render a reconstruction of the physical environment.

Claims (43)

1. A method comprising:

detecting a reflective surface in a digital image captured by a camera that is connected to a machine-detectable tag with encoded unique patterns by:

detecting a reflection of the machine-detectable tag with encoded unique patterns in the digital image;

computing, in response to the detection, a plane that is coincident with a reflective surface associated with the reflection, wherein the plane is computed based on at least the reflection of the machine-detectable tag with encoded unique patterns in the digital image and a relative orientation between the camera and the machine-detectable tag with encoded unique patterns; and

determining a boundary of the reflective surface in the plane based on at least one of a plurality of cues;

storing information associated with the reflective surface, wherein the information comprises a pose of the reflective surface and a boundary of the reflective surface in a three-dimensional model of a physical environment; and

rendering a virtual replica of the physical environment on the reflective surface based on the stored information in the three-dimensional model to reconstruct the physical environment.

2. The method of claim 1 , wherein the camera is associated with a scanning device that captures a series of images of a 3-dimensional area with one or more cameras.

3. The method of claim 2 , wherein the one or more cameras comprise an infrared camera or an RGB camera.

4. The method of claim 1 , wherein detecting the reflection of the machine-detectable tag with encoded unique patterns in the digital image comprises:

identifying candidate locations for the reflection of the machine-detectable tag with encoded unique patterns in the digital image by performing an image processing on the digital image;

determining, for each candidate location, whether a reflection of the machine-detectable tag with encoded unique patterns exists in the location by detecting a reflected image of the machine-detectable tag with encoded unique patterns in the location; and

determining one or more locations of the reflection of the machine-detectable tag with encoded unique patterns within the image, wherein the one or more locations determine a location and an orientation of the reflection of the machine-detectable tag with encoded unique patterns.

5. The method of claim 4 , wherein detecting the reflection of the machine-detectable tag with encoded unique patterns further comprises determining additional information associated with a quality of the detection.

6. The method of claim 4 , wherein each of the one or more locations of the reflection is represented by one or more identifiable points of the reflection of the machine-detectable tag with encoded unique patterns.

7. The method of claim 6 , wherein the one or more identifiable points comprise four corners and a center of the reflection of the machine-detectable tag with encoded unique patterns.

8. The method of claim 1 , wherein computing the plane that is coincident with the reflective surface associated with the reflection comprises:

determining a relative position of the machine-detectable tag with encoded unique patterns from the camera, wherein the position of the machine-detectable tag with encoded unique patterns comprises ordered-positions of four corners and a position of center of the machine-detectable tag with encoded unique patterns; and

computing a plane that bisects each identifiable point of the real machine-detectable tag with encoded unique patterns and a corresponding detected point of the reflected machine-detectable object.

9. The method of claim 8 , wherein the identifiable point comprises any of four corners or a center.

10. The method of claim 1 , wherein the plurality of cues comprises discontinuities, occlusion, geometry, free-space, intensity variance, mean intensity gradient, detection, and average zero-mean normalized cross-correlation (ZNCC).

11. The method of claim 1 , wherein the reflective surface is a fully-reflective surface.

12. The method of claim 1 , wherein the reflective surface is a semi-reflective surface.

13. The method of claim 12 , wherein a reflective surface is determined to be a semi-reflective surface if an intensity of the detected reflection is darker than a threshold intensity.

14. The method of claim 12 , wherein a reflective surface is determined to be a semi-reflective surface if two reflections are detected on the reflective surface, wherein one of the two reflections is reflected at a front of the surface, and wherein the other of the two reflections is reflected at a back of the surface.

15. The method of claim 14 , wherein a thickness of the semi-reflective surface is estimated based on a distance of the two reflections.

16. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

detect a reflective surface in a digital image captured by a camera that is connected to a machine-detectable tag with encoded unique patterns by:

detecting a reflection of the machine-detectable tag with encoded unique patterns in the digital image;

computing, in response to the detection, a plane that is coincident with a reflective surface associated with the reflection, wherein the plane is computed based on at least the reflection of the machine-detectable tag with encoded unique patterns in the digital image and a relative orientation between the camera and the machine-detectable tag with encoded unique patterns; and

determining a boundary of the reflective surface in the plane based on at least one of a plurality of cues;

store information associated with the reflective surface, wherein the information comprises a pose of the reflective surface and a boundary of the reflective surface in a three-dimensional model of a physical environment; and

render a virtual replica of the physical environment on the reflective surface based on the stored information in the three-dimensional model to reconstruct the physical environment.

17. The media of claim 16 , wherein the camera is associated with a scanning device that captures a series of images of a 3-dimensional area with one or more cameras.

18. A system comprising:

one or more processors; and

one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:

detect a reflective surface in a digital image captured by a camera that is connected to a machine-detectable tag with encoded unique patterns by:

detecting a reflection of the machine-detectable tag with encoded unique patterns in the digital image;

computing, in response to the detection, a plane that is coincident with a reflective surface associated with the reflection, wherein the plane is computed based on at least the reflection of the machine-detectable tag with encoded unique patterns in the digital image and a relative orientation between the camera and the machine-detectable tag with encoded unique patterns; and

determining a boundary of the reflective surface in the plane based on at least one of a plurality of cues;

store information associated with the reflective surface, wherein the information comprises a pose of the reflective surface and a boundary of the reflective surface in a three-dimensional model of a physical environment; and

render a virtual replica of the physical environment on the reflective surface based on the stored information in the three-dimensional model to reconstruct the physical environment.

Assignments (1)
CHANGE OF NAME Recorded Jul 6, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060591/0848 →