IP Library › Granted Patent US 10,282,591
Granted Patent B2
US 10,282,591 · App. 14/833,573 · Granted May 7, 2019

Systems and methods for depth map sampling

Inventors: Albrecht Johannes Lindner (La Jolla, CA); Volodimir Slobodyanyuk (San Diego, CA); Stephen Michael Verrall (Carlsbad, CA); Kalin Mitkov Atanassov (San Diego, CA)
Assignee: QUALCOMM Incorporated
G06K9/00208G01S7/4972G01S17/023G01S17/89G06K9/00744G06T7/50H04N13/261G06T2207/10024G06T2207/10028
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,282,591
App. No.
14/833,573
Granted
May 7, 2019
Kind
B2
Abstract

An electronic device is described. The electronic device includes a camera configured to capture an image of a scene. The electronic device also includes an image segmentation mapper configured to perform segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene. The electronic device further includes a memory configured to store the image and the spatial coordinates. The electronic device additionally includes a LIDAR (light+radar) unit, the LIDAR unit steerable to selectively obtain depth values corresponding to at least a subset of the spatial coordinates. The electronic device further includes a depth mapper configured to generate a depth map of the scene based on the depth values and the spatial coordinates.

Claims (43)

1. An electronic device, comprising:

a camera configured to capture an image of a scene;

an image segmentation mapper configured to perform segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene, wherein at least a portion of the image segments comprise non-uniform segments having irregular geometry, size, and distribution based on the image content, and wherein the spatial coordinates of each image segment correspond to a sampling point for each image segment;

a memory configured to store the image and the spatial coordinates;

a LIDAR (light+radar) unit, the LIDAR unit steerable to selectively obtain depth values of the scene corresponding to the spatial coordinates; and

a depth mapper configured to generate a depth map of the scene based on the depth values and the spatial coordinates.

2. The electronic device of claim 1 , wherein the image segmentation mapper is configured to perform segmentation based on an image complexity, wherein a quantity of segments generated is a function of a determined complexity of the image.

3. The electronic device of claim 1 , wherein the LIDAR unit is configured to perform a coarse scan over a region of the scene containing a substantially uniform object within the scene.

4. The electronic device of claim 1 , wherein the sampling point for each image segment corresponds to a centroid of the segment.

5. The electronic device of claim 1 , wherein the image segmentation mapper is configured to provide the sampling point for each image segment to the LIDAR unit and to provide the segments of the image to the depth mapper.

6. The electronic device of claim 1 , wherein the depth mapper is configured to generate the depth map by merging the segments with corresponding depth values.

7. The electronic device of claim 1 , wherein the depth mapper is configured to generate the depth map by populating each segment with a corresponding depth value obtained by the LIDAR unit at the spatial coordinates.

8. The electronic device of claim 1 , wherein the number of depth values obtained by the LIDAR unit is configured to be adjusted based on feedback from a prior depth map.

9. A method, comprising:

capturing an image of a scene;

performing segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene, wherein at least a portion of the image segments comprise non-uniform segments having irregular geometry, size, and distribution based on the image content, and wherein the spatial coordinates of each image segment correspond to a sampling point for each image segment;

obtaining, by a LIDAR unit, depth values of the scene corresponding to the spatial coordinates, wherein the LIDAR unit is steerable to selectively obtain the depth values; and

generating a depth map of the scene based on the depth values and the spatial coordinates.

10. The method of claim 9 , wherein performing segmentation of the image comprises performing segmentation based on an image complexity, wherein a quantity of segments generated is a function of a determined complexity of the image.

11. The method of claim 9 , wherein the LIDAR unit is configured to perform a coarse scan over a region of the scene containing a substantially uniform object within the scene.

12. The method of claim 9 , wherein generating the depth map comprises merging the segments with corresponding depth values.

13. The method of claim 9 , wherein generating the depth map comprises populating each segment with a corresponding depth value obtained by the LIDAR unit at the spatial coordinates.

14. The method of claim 9 , further comprising adjusting the number of depth values obtained by the LIDAR unit based on feedback from a prior depth map.

15. An apparatus, comprising:

means for capturing an image of a scene;

means for performing segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene, wherein at least a portion of the image segments comprise non-uniform segments having irregular geometry, size, and distribution based on the image content, and wherein the spatial coordinates of each image segment correspond to a sampling point for each image segment;

means for obtaining, by a LIDAR unit, depth values of the scene corresponding to the spatial coordinates, wherein the LIDAR unit is steerable to selectively obtain the depth values; and

means for generating a depth map of the scene based on the depth values and the spatial coordinates.

16. The apparatus of claim 15 , wherein the means for performing segmentation of the image comprise means for performing segmentation based on an image complexity, wherein a quantity of segments generated is a function of a determined complexity of the image.

17. The apparatus of claim 15 , wherein the sampling point for each image segment corresponds to a centroid of the segment.

18. The apparatus of claim 15 , wherein the means for generating the depth map comprise means for merging the segments with corresponding depth values.

19. The apparatus of claim 15 , wherein the means for generating the depth map comprise means for populating each segment with a corresponding depth value obtained by the LIDAR unit at the spatial coordinates.

20. The apparatus of claim 15 , further comprising means for adjusting the number of depth values obtained by the LIDAR unit based on feedback from a prior depth map.

21. A computer-program product, comprising a non-transitory tangible computer-readable medium having instructions thereon, the instructions comprising:

code for causing an electronic device to capture an image of a scene;

code for causing the electronic device to perform segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene, wherein at least a portion of the image segments comprise non-uniform segments having irregular geometry, size, and distribution based on the image content, and wherein the spatial coordinates of each image segment correspond to a sampling point for each image segment;

code for causing the electronic device to obtain, by a LIDAR unit, depth values of the scene corresponding to the spatial coordinates, wherein the LIDAR unit is steerable to selectively obtain the depth values; and

code for causing the electronic device to generate a depth map of the scene based on the depth values and the spatial coordinates.

22. The computer-program product of claim 21 , wherein the code for causing the electronic device to perform segmentation of the image comprises code for causing the electronic device to perform segmentation based on an image complexity, wherein a quantity of segments generated is a function of a determined complexity of the image.

23. The computer-program product of claim 21 , wherein the sampling point for each image segment corresponds to a centroid of the segment.

24. The computer-program product of claim 21 , wherein the code for causing the electronic device to generate the depth map comprises code for causing the electronic device to merge the segments with corresponding depth values.

25. The computer-program product of claim 21 , wherein the code for causing the electronic device to generate the depth map comprises code for causing the electronic device to populate each segment with a corresponding depth value obtained by the LIDAR unit at the spatial coordinates.

26. The computer-program product of claim 21 , further comprising code for causing the electronic device to adjust the number of depth values obtained by the LIDAR unit based on feedback from a prior depth map.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 4, 2015
From: LINDNER, ALBRECHT JOHANNES; SLOBODYANYUK, VOLODIMIR; VERRALL, STEPHEN MICHAEL; ATANASSOV, KALIN MITKOV
To: QUALCOMM INCORPORATED
Reel/Frame 036963/0250 →
Continuity (1)
Related Publication 20170061632A1 · Mar 2, 2017
Cited By (5)
US 12,372,654 US 12,524,870 US 12,626,521 US 12,663,518 US 12,736,637