IP Library › Granted Patent US 10,334,151
Granted Patent B2
US 10,334,151 · App. 15/162,426 · Granted Jun 25, 2019

Phase detection autofocus using subaperture images

Inventors: Yuriy Romanenko (San Jose, CA); Edmond Chung (Milpitas, CA); Joachim Deguara (Felton, CA)
Assignee: GOOGLE LLC
H04N5/23212G06K9/6201G06T7/73H04N5/2628
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,334,151
App. No.
15/162,426
Granted
Jun 25, 2019
Kind
B2
Abstract

A light-field camera may have enhanced autofocus functionality that does not require a dedicated optical pathway or dedicated sensors, and yet brings the camera rapidly into focus for image capture. According to one method, light may be received through the aperture of a light-field camera, with the main lens of the light-field camera at a first focus setting. The light may be directed to a sensor of the light-field camera via an optical pathway. The sensor may capture the light to generate preliminary light-field data. In a processor, the preliminary light-field data may be used to generate an offset, which may be used to refocus the main lens to a second focus setting. Additional light may be directed along the same optical pathway and captured with the main lens at the second focus setting to capture a focused light-field image.

Claims (83)

1. A method for focusing a light-field camera, the method comprising:

through an aperture of the light-field camera, receiving first light with a main lens of the light-field camera at a first focus setting;

directing the first light to a sensor of the light-field camera via an optical pathway;

in the sensor, capturing the first light to generate preliminary light-field data;

in a processor, using the preliminary light-field data to generate an offset;

using the offset to refocus the main lens to a second focus setting; and

wherein using the preliminary light-field data to generate the offset comprises:

using the preliminary light-field data to generate a plurality of subaperture images;

computing relative displacements between the subaperture images; and

based on the relative displacements, generating the offset using an averaging of a set of focus offsets.

2. The method of claim 1 , wherein computing relative displacements between the subaperture images comprises:

cropping each of the subaperture images to provide a cropped subaperture image such that all cropped subaperture images depict the same image portion;

identifying at least one feature of each of the cropped subaperture images; and

comparing positions of the features within the cropped subaperture images to determine the relative displacements.

3. The method of claim 2 , wherein calculating a plurality of relative positions of the features for adjacent pairs of the cropped subaperture images yields the set of focus offsets.

4. The method of claim 1 , further comprising:

refocusing the main lens through a sweep from the second focus setting to a third focus setting;

at the sensor, at each of a plurality of intermediate focus settings between the second focus setting and the third focus setting, capturing intermediate light-field data;

at the processor, for each intermediate light-field data, computing a contrast level; and

comparing the contrast levels to identify an optimal intermediate focus setting of the intermediate focus settings at which contrast is the highest.

5. The method of claim 4 , further comprising:

refocusing the main lens to the optimal intermediate focus setting;

through the aperture, receiving second light with the main lens at the optimal intermediate focus setting;

directing the second light to the sensor via the optical pathway; and

in the sensor, capturing the second light to generate a light-field image.

6. The method of claim 1 , further comprising:

at the processor, computing a confidence factor; and

using the confidence factor to determine whether the light-field camera is properly focused.

7. The method of claim 1 , further comprising:

through the aperture, receiving second light with the main lens at the second focus setting;

directing the second light to the sensor via the optical pathway; and

in the sensor, capturing the second light to generate a light-field image.

8. The method of claim 1 , further comprising:

determining whether the light-field camera is properly focused with the main lens at the second focus setting; and

until the light-field camera is properly focused, iterating the steps of:

through the aperture, receiving additional light with the main lens;

directing the additional light to the sensor via the optical pathway;

in the sensor, capturing the additional light to generate additional preliminary light-field data;

in the processor, using the additional preliminary light-field data to generate an additional offset;

using the additional offset to refocus the main lens; and

determining whether the light-field camera is properly focused with the main lens.

9. A light-field camera comprising:

an aperture configured to receive first light with a main lens of the light-field camera at a first focus setting such that the first light is directed to a sensor of the light-field camera via an optical pathway;

a sensor configured to capture the first light to generate preliminary light-field data; and

a processor configured to:

use the preliminary light-field data to generate an offset;

use the offset to cause the main lens to be refocused to a second focus setting; and

wherein the processor is configured to use the preliminary light-field data to generate the offset by:

using the preliminary light-field data to generate a plurality of subaperture images;

computing relative displacements between the subaperture images; and

based on the relative displacements, generating the offset using an averaging of a set of focus offsets.

10. The light-field camera of claim 9 , wherein computing relative displacements between the subaperture images comprises:

cropping each of the subaperture images to provide a cropped subaperture image such that all cropped subaperture images depict the same image portion;

identifying at least one feature of each of the cropped subaperture images; and

comparing positions of the features within the cropped subaperture images to determine the relative displacements.

11. The light-field camera of claim 9 , wherein:

the processor is further configured to cause the main lens to be refocused through a sweep from the second focus setting to a third focus setting;

the sensor is further configured, at each of a plurality of intermediate focus settings between the second focus setting and the third focus setting, to capture intermediate light-field data;

the processor is further configured, for each intermediate light-field data, to compute a contrast level; and

the processor is further configured to compare the contrast levels to identify an optimal intermediate focus setting of the intermediate focus settings at which contrast is the highest.

12. The light-field camera of claim 11 , wherein:

the processor is further configured to cause the main lens to be refocused to the optimal intermediate focus setting;

the aperture is further configured to receive second light with the main lens at the optimal intermediate focus setting such that the second light is directed to the sensor via the optical pathway; and

the sensor is further configured to capture the second light to generate a light-field image.

13. The light-field camera of claim 9 , wherein the processor is further configured to:

computing a confidence factor; and

use the confidence factor to determine whether the light-field camera is properly focused.

14. The light-field camera of claim 9 , wherein:

the aperture is further configured to receive second light with the main lens at the second focus setting such that the second light is directed to the sensor via the optical pathway; and

the sensor is further configured to capture the second light to generate a light-field image.

15. The light-field camera of claim 9 , wherein the processor is further configured to:

determine whether the light-field camera is properly focused with the main lens at the second focus setting; and

until the light-field camera is properly focused, cause the light-field camera to iterate the steps of:

through the aperture, receiving additional light with the main lens such that the additional light is directed to the sensor via the optical pathway;

in the sensor, capturing the additional light to generate additional preliminary light-field data;

using the additional preliminary light-field data to generate an additional offset;

using the additional offset to refocus the main lens; and

determining whether the light-field camera is properly focused with the main lens.

16. The light-field image camera of claim 9 , wherein computing relative displacements between the subaperture images comprises:

cropping each of the subaperture images to provide a cropped subaperture image such that all cropped subaperture images depict the same image portion;

identifying at least one feature of each of the cropped subaperture images;

comparing positions of the features within the cropped subaperture images to determine the relative displacements; and

calculating a plurality of relative position of the features for adjacent pairs of cropped subaperture images to yield the set of focus offsets.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 21, 2019
From: LYTRO, INC.
To: GOOGLE LLC
Reel/Frame 048391/0701 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 29, 2016
From: DEGUARA, JOACHIM
To: LYTRO, INC.
Reel/Frame 039042/0709 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 23, 2016
From: ROMANENKO, YURIY; CHUNG, EDMOND
To: LYTRO, INC.
Reel/Frame 038689/0936 →
Continuity (2)
Continuation In Part 13867333 · Apr 22, 2013
Related Publication 20160269620A1 · Sep 15, 2016