IP Library › Granted Patent US 11,915,502
Granted Patent B2
US 11,915,502 · App. 17/327,309 · Granted Feb 27, 2024

Systems and methods for depth map sampling

Inventors: Albrecht Johannes Lindner (La Jolla, CA); Volodimir Slobodyanyuk (San Diego, CA); Stephen Michael Verrall (Carlsbad, CA); Kalin Mitkov Atanassov (San Diego, CA)
Assignee: QUALCOMM Incorporated
G06V20/647G01S7/4972G01S17/86G01S17/89G06T7/50G06V20/46H04N13/261G06T2207/10024G06T2207/10028
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,915,502
App. No.
17/327,309
Granted
Feb 27, 2024
Kind
B2
Abstract

An electronic imaging device and method for image capture are described. The imaging device includes a camera configured to obtain image information of a scene and that may be focused on a region of interest in the scene. The imaging device also includes a LIDAR unit configured to obtain depth information of at least a portion of the scene at specified scan locations of the scene. The imaging device is configured to detect an object in the scene and provides specified scan locations to the LIDAR unit. The camera is configured to capture an image with an adjusted focus based on depth information, obtained by the LIDAR unit, associated with the detected object.

Claims (39)

1. An imaging device, comprising:

a camera configured to obtain image information of a scene, wherein the camera is capable of autofocusing on a region of interest in the scene;

a LIDAR unit configured to obtain depth information of at least a portion of the scene; and

a processor configured to:

detect an object in the scene using image information obtained from the camera and first depth information obtained from the LIDAR unit;

provide one or more instructions to the LIDAR unit to obtain second depth information at a plurality of specified scan locations, wherein the specified scan locations are characterized by a first scan point density for a first portion of the scene associated with the object and a second scan point density for a second portion of the scene; and

provide one or more instructions to the camera, based on obtained second depth information associated with the specified scan locations associated with the object, to adjust a focus of the camera to the region of interest and capture updated image information of the scene.

2. The imaging device of claim 1 , wherein the processor is further configured to: generate a first image segmentation of the image information associated with the object; and generate a second image segmentation of the image information associated with a portion of the scene not associated with the object.

3. The imaging device of claim 2 , wherein the specified scan locations of the first portion of the scene are based on the first image segmentation, and wherein the specified scan locations of the second portion of the scene are based on the second image segmentation.

4. The imaging device of claim 3 , wherein the first scan point density is greater than the second scan point density.

5. The imaging device of claim 1 , wherein the object is a person or a face.

6. The imaging device of claim 5 , wherein the processor is further configured detect the face using facial recognition.

7. The imaging device of claim 1 , wherein the processor is further configured to generate a segmentation of the image information; and wherein the first scan point density is based on a similarity of image information of neighboring image segments within the object and a similarity of first depth information of neighboring image segments within the object.

8. The imaging device of claim 1 , further comprising a user interface configured to display image information obtained from the camera and capable of receiving user input selection of the object, wherein the processor is further configured to detect the object based on selection information received from the user interface.

9. The imaging device of claim 1 , wherein the processor is further configured to generate a depth map for at least a portion of the scene based on depth information obtained from the LIDAR unit.

10. The imaging device of claim 9 , wherein the processor is further configured to build a three-dimensional model of the object based on the depth map.

11. The imaging device of claim 9 , wherein the processor is further configured to: generate an image segmentation of the image information associated with the object; and generate a refined depth map based on the image segmentation.

12. A method for image capture with an electronic device, comprising:

obtaining image information of a scene from a camera of the electronic device;

obtaining depth information of at least a portion of the scene with a LIDAR unit of the electronic device configured to obtain depth information;

detecting, by a processor, an object in the scene, using the image information obtained from the camera and first depth information obtained from the LIDAR unit;

providing, by the processor, a plurality of specified scan locations to the LIDAR unit to obtain updated depth information, wherein the specified scan locations are characterized by a first scan point density for a first portion of the scene associated with the object and a second scan point density for a second portion of the scene;

providing one or more instructions to the camera, based on the updated depth information associated with the specified scan locations associated with the object, to adjust a focus of the camera to a region of interest associated with the object; and

capturing updated image information of the scene with the adjusted focus.

13. The method of claim 12 , further comprising generating: a first image segmentation of the image information associated with the object; and generating a second image segmentation of the image information associated with a portion of the scene not associated with the object.

14. The method of claim 13 , wherein the specified scan locations of the first portion of the scene are based on the first image segmentation, and wherein the specified scan locations of the second portion of the scene are based on the second image segmentation.

15. The method of claim 14 , wherein the first scan point density is greater than the second scan point density.

16. The method of claim 12 , wherein the object is a person or a face.

17. The method of claim 16 , further comprising detecting, by the processor, the face using facial recognition.

18. The method of claim 12 , further comprising generating, by the processor, an image segmentation of the image information; and wherein the first scan point density is based on a similarity of image information of neighboring image segments within the object and a similarity of first depth information of neighboring image segments within the object.

19. The method of claim 12 , further comprising:

displaying, on a screen, image information obtained from the camera; and

obtaining, with a user interface, object selection information based on the displayed image information, wherein detecting the object by the processor, is based on the object selection information.

20. A non-transitory computer readable storage medium having stored thereon instructions that, when executed by a processor of a device that includes a camera and a LIDAR unit, cause the device to:

obtain image information of a scene from the camera;

obtain first depth information of at least a portion of the scene with the LIDAR unit configured to obtain depth information;

detect an object in the scene using the image information obtained from the camera and first depth information obtained from the LIDAR unit;

obtain updated depth information from the LIDAR unit at a plurality of specified scan locations, wherein the plurality of specified scan locations are characterized by a first scan point density for a first portion of the scene associated with the object and a second scan point density for a second portion of the scene, and wherein the first scan point density is greater than the second scan point density; and

focus the camera based on the updated depth information of the second plurality of specified scan locations associated with the object and capture updated image information of the scene.

Assignments (3)
CORRECTIVE ASSIGNMENT TO CORRECT THE 3ND AND 3RD INVENTOR'S COUNTRY IN THE ASSIGNMENT DOCUMENT PREVIOUSLY RECORDED AT REEL: 059170 FRAME: 0097. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT . Recorded Mar 22, 2022
From: LINDNER, ALBRECHT JOHANNES; SLOBODYANYUK, VOLODIMIR; VERRALL, STEPHEN MICHAEL; ATANASSOV, KALIN MITKOV
To: QUALCOMM INCORPORATED
Reel/Frame 059476/0334 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 4, 2022
From: LINDNER, ALBRECHT JOHANNES; SLOBODYANYUK, VOLODIMIR; VERRALL, STEPHEN MICHAEL; ATANASSOV, KALIN MITKOV
To: QUALCOMM INCORPORATED
Reel/Frame 059170/0097 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 21, 2021
From: LIDNER, ALBRECHT JOHANNES; SLOBODYANYUK, VOLODIMIR; VERRALL, STEPHEN MICHAEL; ATANASSOV, KALIN MITKOV
To: QUALCOMM INCORPORATED
Reel/Frame 056317/0803 →
Continuity (3)
Continuation 16359441 · Mar 20, 2019
Continuation 14833573 · Aug 24, 2015
Related Publication 20210279444A1 · Sep 9, 2021
Cited By (1)
US 12,437,579