IP Library Granted Patent US 10,636,155
Granted Patent B2
US 10,636,155 · App. 16/180,039 · Granted Apr 28, 2020

Multi-modal depth mapping

Inventors: Alexander Shpunt (Portola Valley, CA); Gerard Medioni (Los Angeles, CA); Daniel Cohen (Tel Aviv, IL); Erez Sali (Savion, IL); Ronen Deitch (Modiin, IL)
Assignee: APPLE INC.
G06T7/521G01B11/22G01B11/25G06T7/593H04N5/33H04N13/239H04N13/25H04N13/254H04N13/271H04N13/296G06T2207/10024G06T2207/10028
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,636,155
App. No.
16/180,039
Granted
Apr 28, 2020
Kind
B2
Abstract

A method for depth mapping includes acquiring first depth data with respect to an object using a first depth mapping technique and providing first candidate depth coordinates for a plurality of pixels, and acquiring second depth data with respect to the object using a second depth mapping technique, different from the first depth mapping technique, and providing second candidate depth coordinates for the plurality of pixels. A weighted voting process is applied to the first and second depth data in order to select one of the candidate depth coordinates at each pixel. A depth map of the object is output, including the selected one of the candidate depth coordinates at each pixel.

Claims (33)

1. A method for depth mapping, comprising:

acquiring first depth data with respect to an object using a first depth mapping technique and providing first candidate depth coordinates for a plurality of pixels;

acquiring second depth data with respect to the object using a second depth mapping technique, different from the first depth mapping technique, and providing second candidate depth coordinates for the plurality of pixels,

wherein acquiring the first and second depth data comprises computing respective measures of confidence associated with the first and second candidate depth coordinates;

applying a weighted voting process to the first and second depth data in order to select one of the candidate depth coordinates at each pixel,

wherein applying the weighted voting process comprises weighting votes for the candidate depth coordinates responsively to the respective measures of confidence; and

outputting a depth map of the object comprising the selected one of the candidate depth coordinates at each pixel.

2. The method according to claim 1 , wherein the first and second candidate depth coordinates comprise, for at least some of the pixels, a null coordinate indicating that no valid depth coordinate was found.

3. The method according to claim 1 , wherein applying the weighted voting process comprises applying weighted tensor voting among the pixels in order to select the one of the candidate depth coordinates based on the candidate depth coordinates at neighboring pixels.

4. The method according to claim 3 , wherein applying the weighted tensor voting comprises computing a weighted sum of covariance matrices over the neighboring pixels, and selecting the one of the candidate depth coordinates based on a difference between eigenvalues of the summed covariance matrices.

5. The method according to claim 1 , wherein at least one of the first and second depth mapping techniques comprises pattern-based depth mapping.

6. The method according to claim 1 , wherein at least one of the first and second depth mapping techniques comprises stereoscopic depth mapping.

7. The method according to claim 6 , wherein the first depth mapping technique comprises pattern-based depth mapping, while the second depth mapping technique comprises stereoscopic depth mapping, and

wherein acquiring the first depth data comprises projecting a pattern of optical radiation onto the object, capturing a first image of the pattern on the object using a first image sensor, and processing the first image to generate pattern-based depth data with respect to the object, and

wherein acquiring the second depth data comprises capturing a second image of the object using a second image sensor, and processing the second image together with the first image to generate stereoscopic depth data with respect to the object.

8. Apparatus for depth mapping, comprising:

one or more sensors, which are configured to acquire first depth data with respect to an object using a first depth mapping technique and providing first candidate depth coordinates for a plurality of pixels, and to acquire second depth data with respect to the object using a second depth mapping technique, different from the first depth mapping technique, and providing second candidate depth coordinates for the plurality of pixels; and

a processor, which is configured to apply a weighted voting process to the first and second depth data in order to select one of the candidate depth coordinates at each pixel, and to output a depth map of the object comprising the selected one of the candidate depth coordinates at each pixel,

wherein the processor is configured to associate respective measures of confidence with the first and second candidate depth coordinates, and to weight votes for the candidate depth coordinates responsively to the respective measures of confidence.

9. The apparatus according to claim 8 , wherein the first and second candidate depth coordinates comprise, for at least some of the pixels, a null coordinate indicating that no valid depth coordinate was found.

10. The apparatus according to claim 8 , wherein the processor is configured to apply weighted tensor voting among the pixels in order to select the one of the candidate depth coordinates based on the candidate depth coordinates at neighboring pixels.

11. The apparatus according to claim 10 , wherein the processor is configured to compute a weighted sum of covariance matrices over the neighboring pixels, and to select the one of the candidate depth coordinates based on a difference between eigenvalues of the summed covariance matrices.

12. The apparatus according to claim 8 , wherein at least one of the first and second depth mapping techniques comprises pattern-based depth mapping.

13. The apparatus according to claim 8 , wherein at least one of the first and second depth mapping techniques comprises stereoscopic depth mapping.

14. The apparatus according to claim 13 , wherein the first depth mapping technique comprises pattern-based depth mapping, while the second depth mapping technique comprises stereoscopic depth mapping, and wherein the apparatus comprises:

an illumination subassembly, which is configured to project a pattern of optical radiation onto an object;

a first image sensor, which is configured to capture a first image of the pattern on the object; and

at least a second image sensor, which is configured to capture at least a second image of the object,

wherein the processor is configured to process the first image to generate pattern-based depth data with respect to the object and to process the second image together with the first image to generate stereoscopic depth data with respect to the object.

15. A computer software product, comprising a computer-readable medium in which program instructions are stored, which instructions, when read by a processor, cause the processor to acquire first depth data with respect to an object using a first depth mapping technique and providing first candidate depth coordinates for a plurality of pixels, and to acquire second depth data with respect to the object using a second depth mapping technique, different from the first depth mapping technique, and providing second candidate depth coordinates for the plurality of pixels, to apply a weighted voting process to the first and second depth data in order to select one of the candidate depth coordinates at each pixel, and to output a depth map of the object comprising the selected one of the candidate depth coordinates at each pixel,

wherein the instructions cause the processor to associate respective measures of confidence with the first and second candidate depth coordinates, and to weight votes for the candidate depth coordinates responsively to the respective measures of confidence.

16. The product according to claim 15 , wherein the first and second candidate depth coordinates comprise, for at least some of the pixels, a null coordinate indicating that no valid depth coordinate was found.

17. The product according to claim 15 , wherein the instructions cause the processor to apply weighted tensor voting among the pixels in order to select the one of the candidate depth coordinates based on the candidate depth coordinates at neighboring pixels.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 5, 2018
From: SHPUNT, ALEXANDER; MEDIONI, GERARD; COHEN, DANIEL; SALI, EREZ; DEITCH, RONEN
To: APPLE INC.
Reel/Frame 047407/0231 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 5, 2018
From: SHPUNT, ALEXANDER; MEDIONI, GERARD; COHEN, DANIEL; SALI, EREZ; DEITCH, RONEN
To: PRIMESENSE LTD.
Reel/Frame 047407/0236 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 5, 2018
From: PRIMESENSE LTD.
To: APPLE INC.
Reel/Frame 047407/0241 →
Continuity (4)
Continuation 15271398 · Sep 21, 2016
Continuation 12844864 · Jul 28, 2010
Provisional Application 61229754 · Jul 30, 2009
Related Publication 20190087969A1 · Mar 21, 2019
Cited By (1)
US 12,590,798