IP Library Granted Patent US 11,682,107
Granted Patent B2
US 11,682,107 · App. 16/220,434 · Granted Jun 20, 2023

Depth of field adjustment in images based on time of flight depth maps

Inventor: Peter Shintani (San Diego, CA)
Assignee: SONY CORPORATION
H04N13/271G01S17/32G01S17/89G06F3/013G06T5/002G06V10/25
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,682,107
App. No.
16/220,434
Granted
Jun 20, 2023
Kind
B2
Abstract

An image capturing apparatus and a method for depth of filed (DOF) adjustment in images based on time of flight (TOF) depth maps is provided. The image capturing apparatus includes an image sensor and circuitry. The circuitry generates a TOF depth map of a scene that includes a plurality of objects. The TOF depth map includes information associated with distances between the image sensor and surfaces of the plurality of objects. The circuitry divides the TOF depth map into a plurality of regions that corresponds to at least one object of the plurality of objects. The circuitry determines a region of interest from the plurality of regions and adjusts the DOF of the at least one object associated with the determined region of interest. The circuitry further controls the image sensor to capture an image of the scene based on the adjusted DOF.

Claims (64)

1. An image capturing apparatus, comprising:

an image sensor;

at least one illuminator; and

circuitry configured to:

control the at least one illuminator to emit first light signals to irradiate a plurality of objects in a scene;

control the image sensor to receive second light signals that correspond to a reflection of the emitted first light signals from surfaces of the plurality of objects in the scene;

calculate a Time of flight (TOF) of the emitted first light signals based on the emitted first light signals and the received second light signals;

calculate distances between the image sensor and the surfaces of the plurality of objects based on the calculated TOF;

generate, based on the calculated distances, a TOF depth map of the scene that includes the plurality of objects, wherein the TOF depth map includes information associated with the distances between the image sensor and the surfaces of the plurality of objects;

divide the generated TOF depth map into a plurality of regions, wherein each of the plurality of regions includes at least one object of the plurality of objects;

determine a region of interest from the plurality of regions;

adjust a Depth of field (DOF) of the at least one object from the plurality of objects, wherein the at least one object corresponds to the region of interest in the generated TOF depth map; and

control the image sensor to capture an image of the scene, wherein the capture of the image is based on the adjusted DOF.

2. The image capturing apparatus according to claim 1 , wherein

the circuitry is further configured to blur a portion of the captured image,

the blurred portion corresponds to a region of the plurality of regions, and

the region is different from the region of interest.

3. The image capturing apparatus according to claim 1 , wherein the region of interest corresponds to a Plane of Focus (POF) of the image sensor.

4. The image capturing apparatus according to claim 1 , further comprising an eye movement detector circuit,

wherein the circuitry is further configured to:

control the eye movement detector circuit to track an eye movement of a user associated with the image capturing apparatus; and

determine the region of interest based on the tracked eye movement of the user.

5. The image capturing apparatus according to claim 1 , further comprising:

at least one lens; and

a lens actuator,

wherein the circuitry is further configured to control, based on the determined region of interest, the lens actuator to adjust a position of the at least one lens with respect to a focal plane of the image sensor.

6. The image capturing apparatus according to claim 5 , further comprising an aperture size controller, wherein

the circuitry is further configured to:

control, based on the determined region of interest, the aperture size controller to set an aperture size of an aperture stop of the image capturing apparatus; and

adjust the DOF of the at least one object based on the set aperture size and the adjusted position of the at least one lens.

7. The image capturing apparatus according to claim 1 , wherein the region of interest corresponds to a center position of the scene.

8. The image capturing apparatus according to claim 1 , wherein the circuitry is further configured to:

receive a user input; and

determine the region of interest from the plurality of regions based on the received user input.

9. The image capturing apparatus according to claim 1 , wherein the circuitry is further configured to:

detect the plurality of objects based on the generated TOF depth map; and

divide the generated TOF depth map into the plurality of regions based on the detection of the plurality of objects.

10. The image capturing apparatus according to claim 1 , wherein the circuitry is further configured to divide the generated TOF depth map into the plurality of regions based on a size of each of the plurality of objects in the generated TOF depth map.

11. A method, comprising:

in an image capturing apparatus that includes an image sensor and at least one illuminator:

controlling the at least one illuminator to emit first light signals to irradiate a plurality of objects in a scene;

controlling the image sensor to receive second light signals that correspond to a reflection of the emitted first light signals from surfaces of the plurality of objects in the scene;

calculating a Time of flight (TOF) of the emitted first light signals based on the emitted first light signals and the received second light signals;

calculating distances between the image sensor and the surfaces of the plurality of objects based on the calculated TOF;

generating, based on the calculated distances, a TOF depth map of the scene that includes the plurality of objects, wherein the TOF depth map includes information associated with the distances between the image sensor and the surfaces of the plurality of objects;

dividing the generated TOF depth map into a plurality of regions, wherein each of the plurality of regions includes at least one object of the plurality of objects;

determining a region of interest from the plurality of regions;

adjusting a Depth of field (DOF) of the at least one object from the plurality of objects, wherein the at least one object corresponds to the region of interest in the generated TOF depth map; and

controlling the image sensor to capture an image of the scene, wherein the capture of the image is based on the adjusted DOF.

12. The method according to claim 11 , further comprising blurring a portion of the captured image, wherein

the blurred portion corresponds to a region of the plurality of regions, and

the region is different from the region of interest.

13. The method according to claim 11 , wherein the region of interest corresponds to a Plane of Focus (POF) of the image sensor.

14. The method according to claim 11 , further comprising:

tracking an eye movement of a user associated with the image capturing apparatus; and

determining the region of interest based on the tracked eye movement of the user.

15. The method according to claim 11 , further comprising controlling, based on the determined region of interest, a lens actuator of the image capturing apparatus to adjust a position of at least one lens of the image capturing apparatus with respect to a focal plane of the image sensor.

16. The method according to claim 15 , further comprising:

controlling, based on the determined region of interest, an aperture size controller to set an aperture size of an aperture stop of the image capturing apparatus; and

adjusting the DOF of the at least one object based on the set aperture size and the adjusted position of the at least one lens.

17. The method according to claim 11 , wherein the region of interest corresponds to a center position of the scene.

18. The method according to claim 11 , further comprising:

receiving a user input; and

determining the region of interest from the plurality of regions based on the received user input.

Assignments (2)
CHANGE OF NAME Recorded May 16, 2023
From: SONY CORPORATION
To: SONY GROUP CORPORATION
Reel/Frame 063664/0882 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 17, 2018
From: SHINTANI, PETER
To: SONY CORPORATION
Reel/Frame 047793/0988 →
Continuity (1)
Related Publication 20200195909A1 · Jun 18, 2020