IP Library Granted Patent US 10,877,622
Granted Patent B2
US 10,877,622 · App. 16/116,505 · Granted Dec 29, 2020

Detection of structured light for depth sensing

Inventors: Nitay Romano (Sunnyvale, CA); Ricardo Garcia (San Mateo, CA); Nadav Grossinger (Foster City, CA); Robert Hasbun (San Jose, CA); Niv Kantor (San Francisco, CA)
Assignee: Facebook Technologies, LLC
G06F3/04815G01B11/25G02B27/0172G06K9/00671G06T19/006H04N9/31H04N13/344
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,877,622
App. No.
16/116,505
Granted
Dec 29, 2020
Kind
B2
Abstract

In one embodiment, a computing system may access a first image and a second image of at least a common portion of an environment while a light emission with a predetermined emission pattern is projected by a projector. The first and second images are respectively captured by a first and a second detector that are respectively separated from the projector by a first and a second distance. The system may determine that a first portion of the first image corresponds to a second portion of the second image. The system may compute, using triangulation, a first depth value associated with the first portion and a second depth value associated with the second portion. The system may determine that the first and second depth values match in accordance with one or more predetermined criteria, and generate a depth map of the environment based on at least one of the depth values.

Claims (48)

1. A system comprising:

a projector electronic device configured to project a light emission with a predetermined emission pattern into an environment;

a first detector electronic device and a second detector electronic device configured to respectively capture a first image and a second image of at least a common portion of the environment while the light emission is projected, wherein the first detector electronic device and the projector electronic device are separated by a first predetermined distance and the second detector electronic device and the projector electronic device are separated by a second predetermined distance; and

at least one processor configured to:

determine that a first portion of the first image corresponds to a second portion of the second image;

compute a first depth value associated with the first portion of the first image based on a first triangulation calculation corresponding to a first triangle, the first triangle having corners corresponding to the first detector electronic device, the projector electronic device, and a portion of an object depicted in the first portion of the first image, and the first triangle being formed using at least a location of the first portion within the first image, the predetermined emission pattern, and the first predetermined distance between the first detector electronic device and the projector electronic device;

compute a second depth value associated with the second portion of the second image based on a second triangulation calculation corresponding to a second triangle, the second triangle having corners corresponding to the second detector electronic device, the projector electronic device, and a portion of the object depicted in the second portion of the second image, and the second triangle being formed using at least a location of the second portion within the second image, the predetermined emission pattern, and the second predetermined distance between the second detector electronic device and the projector electronic device; and

generate a depth map of the environment based on at least one of the first depth value or the second depth value.

2. The system of claim 1 , wherein the at least one processor is further configured to disregard the first depth value and the second depth value in the generation of the depth map when the first depth value and the second depth value fail to match in accordance with one or more predetermined criteria.

3. The system of claim 1 , wherein the projector electronic device, the first detector electronic device, and the second detector electronic device are part of a head-mounted display.

4. The system of claim 3 , wherein the first detector electronic device and the second detector electronic device are unfiltered cameras.

5. The system of claim 4 , wherein the at least one processor is further configured to identify objects in the environment based on at least the first image or the second image.

6. The system of claim 1 , wherein the predetermined emission pattern comprises two or more parallel lines.

7. The system of claim 6 , wherein the two or more parallel lines are oriented diagonally.

8. The system of claim 1 , wherein the first detector electronic device and the second detector electronic device are configured to capture the first image and the second image at a same time.

9. The system of claim 1 , wherein, in accordance with one or more predetermined criteria, the first depth value and the second depth value match if a difference between the first depth value and the second depth value is within a predetermined threshold.

10. A method comprising, by a computing system:

accessing a first image and a second image of at least a common portion of an environment while a light emission with a predetermined emission pattern is projected by a projector electronic device, wherein the first image and the second image are respectively captured by a first detector electronic device and a second detector electronic device, and wherein the first detector electronic device and the projector electronic device are separated by a first predetermined distance and the second detector electronic device and the projector electronic device are separated by a second predetermined distance;

determining that a first portion of the first image corresponds to a second portion of the second image;

computing a first depth value associated with the first portion of the first image based on a first triangulation calculation corresponding to a first triangle, the first triangle having corners corresponding to the first detector electronic device, the projector electronic device, and a portion of an object depicted in the first portion of the first image, and the first triangle being formed using at least a location of the first portion within the first image, the predetermined emission pattern, and the first predetermined distance between the first detector electronic device and the projector electronic device;

computing a second depth value associated with the second portion of the second image based on a second triangulation calculation corresponding to a second triangle, the second triangle having corners corresponding to the second detector electronic device, the projector electronic device, and a portion of the object depicted in the second portion of the second image, and the second triangle being formed using at least a location of the second portion within the second image, the predetermined emission pattern, and the second predetermined distance between the second detector electronic device and the projector electronic device; and

generating a depth map of the environment based on at least one of the first depth value or the second depth value.

11. The method of claim 10 , further comprising:

determining that a third portion of the first image corresponds to a fourth portion of the second image;

computing a third depth value associated with the third portion of the first image;

computing a fourth depth value associated with the fourth portion of the second image; and

determining that the third depth value and the fourth depth value fail to match in accordance with one or more predetermined criteria;

wherein the third depth value and the fourth depth value are disregarded in the generation of the depth map.

12. The method of claim 10 , wherein the projector electronic device, the first detector electronic device, and the second detector electronic device are part of a head-mounted display.

13. The method of claim 12 , wherein the first detector electronic device and the second detector electronic device are unfiltered cameras.

14. The method of claim 13 , further comprising:

identifying objects in the environment based on at least the first image or the second image.

15. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

access a first image and a second image of at least a common portion of an environment while a light emission with a predetermined emission pattern is projected by a projector electronic device, wherein the first image and the second image are respectively captured by a first detector electronic device and a second detector electronic device, and wherein the first detector electronic device and the projector electronic device are separated by a first predetermined distance and the second detector electronic device and the projector electronic device are separated by a second predetermined distance;

determine that a first portion of the first image corresponds to a second portion of the second image;

compute a first depth value associated with the first portion of the first image based on a first triangulation calculation corresponding to a first triangle, the first triangle having corners corresponding to the first detector electronic device, the projector electronic device, and a portion of an object depicted in the first portion of the first image, and the first triangle being formed using at least a location of the first portion within the first image, the predetermined emission pattern, and the first predetermined distance between the first detector electronic device and the projector electronic device;

compute a second depth value associated with the second portion of the second image based on a second triangulation calculation corresponding to a second triangle, the second triangle having corners corresponding to the second detector electronic device, the projector electronic device, and a portion of the object depicted in the second portion of the second image, and the second triangle being formed using at least a location of the second portion within the second image, the predetermined emission pattern, and the second predetermined distance between the second detector electronic device and the projector electronic device; and

generate a depth map of the environment based on at least one of the first depth value or the second depth value.

16. The media of claim 15 , wherein the software is further operable when executed to:

determine that a third portion of the first image corresponds to a fourth portion of the second image;

compute a third depth value associated with the third portion of the first image;

compute a fourth depth value associated with the fourth portion of the second image; and

determine that the third depth value and the fourth depth value fail to match in accordance with one or more predetermined criteria;

wherein the third depth value and the fourth depth value are disregarded in the generation of the depth map.

17. The media of claim 15 , wherein the projector electronic device, the first detector electronic device, and the second detector electronic device are part of a head-mounted display.

18. The media of claim 17 , wherein the first detector electronic device and the second detector electronic device are unfiltered cameras.

19. The media of claim 18 , wherein the software is further operable when executed to:

identify objects in the environment based on at least the first image or the second image.

Assignments (3)
CHANGE OF NAME Recorded Jul 6, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060591/0848 →
CHANGE OF NAME Recorded Nov 11, 2020
From: OCULUS VR, LLC
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 054386/0170 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 30, 2018
From: ROMANO, NITAY; KANTOR, NIV; GARCIA, RICARDO; HASBUN, ROBERT; GROSSINGER, NADAV
To: OCULUS VR, LLC
Reel/Frame 047354/0300 →
Cited By (2)
US 12,361,661 US 12,455,162