IP Library Granted Patent US 9,191,578
Granted Patent B2
US 9,191,578 · App. 13/538,029 · Granted Nov 17, 2015

Enhanced image processing with lens motion

Inventors: Ilya Blayvas (Ramat Gan, IL); Noam Sorek (Zichron Yacoov, IL); Ilia Vitsnudel (Even Yehoda, IL)
Assignee: Broadcom Corporation
H04N5/2356G02B7/38G02B27/0075H04N13/0271H04N5/23212H04N13/0221
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,191,578
App. No.
13/538,029
Granted
Nov 17, 2015
Kind
B2
Abstract

An exemplary imaging system operates by capturing image information as a lens moves within a defined focal length range during image exposure, where a focused image having an extended depth of field can be generated by using a profile of lens motion in processing the image.

Claims (52)

1. An imaging system comprising:

an imaging lens that is adjustable to different focal lengths;

an image sensor comprising a plurality of shutters and configured to receive image information from the imaging lens;

a mechanical mechanism to change a lens focal position during duration of a single image frame exposure of the image sensor; and

at least one hardware processing circuit configured to:

obtain a distance map of a scene from at least a first and second image captured by the image sensor while changing the lens focal position during an exposure of the first and second images, the lens focal position being changed with a first speed profile for the first image and a second speed profile for the second image, wherein the first and second speed profiles are asymmetric and associated with non-constant acceleration;

analyze the obtained distance map to identify first regions of the scene that are in focus and second regions of the scene that are out of focus;

construct a point spread function based at least on analyzing the obtained distance map;

capture a third image during a single image frame exposure of the image sensor by activating one or more of the plurality of shutters of the image sensor based on the distance map, the third image being captured subsequent to the first and second images; and

deconvolve the third image using the constructed point spread function to obtain an in-focus image.

2. The imaging system of claim 1 , wherein for the first, the lens focal position is continually changed at a first varying rate of speed during the exposure of the first image, wherein for the second image, the lens focal position is continually changed at a different varying rate of speed in an opposite direction during the exposure of the second image.

3. The imaging system of claim 1 , wherein each of the plurality of shutters is associated with a pixel of the image sensor, wherein each of the plurality of shutters is synchronized with the changing lens focal position of the imaging lens and is activated at a set lens focal position for a particular pixel to produce an in-focus capture of pixel data.

4. The imaging system of claim 1 , wherein the at least one hardware processing circuit is further configured to determine an autofocus setting to be used to capture a subsequent image scene at a particular distance from the imaging system to an object by capturing two image frames comprising a fourth image and a fifth image, wherein for the fourth image, the lens focal position is continually changed at a first varying rate of speed during exposure of the fourth image, wherein for the fifth image, the lens focal position is continually changed at a different varying rate of speed during exposure of the fifth image, wherein a comparison of the image information captured during the fourth image and the fifth image identifies lens focal positions to be used as autofocus settings for varying distances from the imaging system to objects in the subsequent image scene.

5. An imaging method comprising:

capturing, by an image sensor, image data comprising a first and a second image while changing a lens focal position of a lens during an exposure of the first and second images, the lens focal position of the lens being changed with a first speed profile for the first image and a second speed profile for the second image, wherein the first and second speed profiles are asymmetric and associated with non-constant acceleration;

obtaining a distance map of a scene from at least the first and second images;

analyzing the obtained distance map to identify first regions of the scene that are in focus and second regions of the scene that are out of focus;

constructing a point spread function based at least on analyzing the obtained distance map;

capturing a third image during a single image frame exposure of the image sensor by activating one or more of a plurality of shutters of the image sensor based on the distance map, the third image being captured subsequent to the first and second images; and

deconvolving the third image using the constructed point spread function to obtain an in-focus image.

6. The imaging method of claim 5 , wherein:

the first image is captured as the lens is moved within a defined first focal sweep, wherein the defined first focal sweep utilizes the first speed profile; and

the second image is captured as the lens is moved within a defined second focal sweep, wherein the defined second focal sweep utilizes the second speed profile, the defined second focal sweep including a same range of lens focal positions as the defined first focal sweep.

7. The imaging method of claim 5 , further comprising generating a stereoscopic image by combining distance map information with the in-focus image.

8. The imaging method of claim 5 , further comprising:

storing a lens focal position associated with a distance from the lens to an object in the scene corresponding to an identified sharpest focus depth position as an autofocus setting; and

in a subsequent image capture, using the stored autofocus setting to capture a second scene of an object at a same distance from the lens to an object in the scene.

9. The imaging method of claim 5 , further comprising synchronizing activation of the plurality of shutters with movement of the lens.

10. A non-transitory computer readable medium storing instructions executable by a processor, the instructions comprising:

instructions to capture image data of a scene by an image sensor during an exposure of a first image as a lens focal position of a lens is being changed with a first speed profile;

instructions to capture image data of the scene during an exposure of a second image as the lens focal position of the lens is being changed with a second speed profile, wherein the first and second speed profiles are asymmetric and associated with non-constant acceleration;

instructions to obtain a distance map of the scene from at least the first and second images;

instructions to analyze the obtained distance map to identify first regions of the scene that are in focus and second regions of the scene that are out of focus;

instructions to construct a point spread function based at least on analyzing the obtained distance map;

instructions to capture a third image during a single image frame exposure of the image sensor by activating one or more of a plurality of shutters of the image sensor based on the distance map, the third image being captured subsequent to the first and second images; and

instructions to deconvolve, using the constructed point spread function, the third image to obtain an in-focus image.

11. The computer readable medium of claim 10 , the instructions further comprising:

instructions to store a lens focal position associated with a distance from the lens to an object in the scene corresponding to an identified sharpest focus depth position as an autofocus setting; and

instructions to utilize, in a subsequent image capture, the stored autofocus setting to capture a second scene of an object at a same distance from the lens to an object in the scene.

12. The computer readable medium of claim 10 , the instructions further comprising instructions to identify depth positions of pixels at respective sharpest lens focal positions, wherein the obtained distance map is based at least in part on the identified depth positions of the pixels.

13. The computer readable medium of claim 10 , the instructions further comprising instructions to synchronize activation of the plurality of shutters with movement of the lens.

14. The imaging system of claim 1 , wherein the at least one hardware processing circuit is further configured to synchronize activation of the plurality of shutters with movement of the imaging lens.

15. The imaging system of claim 1 , wherein:

the imaging sensor is configured to capture the first image when the imaging lens is moved within a defined first focal sweep, wherein the defined first focal sweep utilizes the first speed profile;

the imaging sensor is configured to capture the second image when the imaging lens is moved within a defined second focal sweep, wherein the defined second focal sweep utilizes the second speed profile, the defined second focal sweep including a same range of lens focal positions as the defined first focal sweep; and

the at least one hardware processing circuit is configured to compare the first image and the second image to identify sharpest focus depth positions of pixels in the range of the lens focal positions, wherein the obtained distance map is based at least in part on the sharpest focus depth positions of the pixels.

16. The imaging system of claim 1 , wherein the at least one hardware processing circuit is further configured to generate a stereoscopic image by combining the distance map information with the in-focus image.

17. The imaging method of claim 5 , further comprising:

comparing the first image and the second image to identify sharpest focus depth positions of pixels in a range of the lens focal positions for the scene, wherein the obtained distance map is based at least in part on the sharpest focus depth positions of the pixels.

18. The imaging method of claim 17 , wherein the sharpest depth positions of the pixels are identified based at least on evaluating content of the scene.

19. The computer readable medium of claim 10 , the instructions further comprising instructions to compare the first image and the second image to identify sharpest focus depth positions of pixels in a range of the lens focal positions, wherein the obtained distance map is based at least in part on the sharpest focus depth positions of the pixels.

20. The computer readable medium of claim 10 , the instructions further comprising instructions to generate a stereoscopic image by combining distance map information with the in-focus image.

Assignments (7)
CORRECTIVE ASSIGNMENT TO CORRECT THE PATENT NUMBER 9,385,856 TO 9,385,756 PREVIOUSLY RECORDED AT REEL: 47349 FRAME: 001. ASSIGNOR(S) HEREBY CONFIRMS THE MERGER. Recorded Mar 22, 2019
From: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
To: AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE. LIMITED
Reel/Frame 051144/0648 →
CORRECTIVE ASSIGNMENT TO CORRECT THE EFFECTIVE DATE PREVIOUSLY RECORDED ON REEL 047229 FRAME 0408. ASSIGNOR(S) HEREBY CONFIRMS THE THE EFFECTIVE DATE IS 09/05/2018. Recorded Oct 29, 2018
From: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
To: AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE. LIMITED
Reel/Frame 047349/0001 →
MERGER Recorded Oct 4, 2018
From: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
To: AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE. LIMITED
Reel/Frame 047229/0408 →
TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS Recorded Feb 3, 2017
From: BANK OF AMERICA, N.A., AS COLLATERAL AGENT
To: BROADCOM CORPORATION
Reel/Frame 041712/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 1, 2017
From: BROADCOM CORPORATION
To: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
Reel/Frame 041706/0001 →
PATENT SECURITY AGREEMENT Recorded Feb 11, 2016
From: BROADCOM CORPORATION
To: BANK OF AMERICA, N.A., AS COLLATERAL AGENT
Reel/Frame 037806/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 13, 2012
From: BLAYVAS, ILYA; SOREK, NOAM; VITSNUDEL, ILIA
To: BROADCOM CORPORATION
Reel/Frame 028774/0737 →
Continuity (1)
Related Publication 20140002606A1 · Jan 2, 2014