Automatic generation of all-in-focus images with a mobile camera
The present disclosure describes systems and techniques directed to producing an all-in-focus image with a camera of a mobile device, in particular, cameras with shallow depth-of-field. User equipment includes a sensor for determining distance to an object in a camera's field-of-view. Based on a depth map of the field-of-view, a plurality of segments is inferred, each segment defining a unique focus area within the camera's field-of-view. An autofocus lens of the camera sweeps to a respective focal distance associated with each of the plurality of segments. The camera captures sample images at each focal distance swept by the autofocus lens. The user equipment produces an all-in-focus image by combining or merging portions of the captured sample images.
1. A method comprising:
capturing a plurality of buffer images of a field-of-view of a camera;
storing the buffer images at an image buffer;
determining a set of focal distances between the camera and objects-of-interest in the field-of-view;
inferring a plurality of segments in the field-of-view, each segment of the plurality of segments corresponding to a focal distance of the set of focal distances and defining a focus area within the field-of-view of the camera, the focus area corresponding with objects-of-interest at similar focal distances from the camera;
determining whether a buffer image of the buffer images captures the field-of-view of the camera at any of the focal distances in the set of focal distances;
responsive to determining that a buffer image captures the field-of-view of the camera at a first focal distance in the set of focal distances, sweeping an autofocus lens of the camera to one or more of the focal distances from the set of focal distances other than the first focal distance;
capturing sample images at one or more of the focal distances from the set of focal distances swept by the autofocus lens;
combining at least one of the sample images captured at the one or more focal distances swept by the autofocus lens with another image to produce an all-in-focus image; and
outputting, for display, an indication of the all-in-focus image.
2. The method of claim 1 , wherein combining at least one of the sample images captured at the one or more focal distances swept by the autofocus lens with the other image to produce the all-in-focus image further comprises:
combining one of the sample images with one of the buffer images.
3. The method of claim 1 , wherein combining at least one of the sample images captured at the one or more focal distances swept by the autofocus lens with the other image to produce the all-in-focus image further comprises:
combining a second sample image with the other image to produce the all-in-focus image, wherein the second sample image is different than the at least one of the sample images being combined to produce the all-in-focus image.
4. The method of claim 1 , further comprising:
obtaining sensor data, from at least one of a depth sensor, a contrast sensor, or a phase-detection-sensor.
5. The method of claim 4 , wherein determining a set of focal distances between the camera and objects-of-interest in the field-of-view is performed utilizing the sensor data.
6. The method of claim 5 , wherein determining, based on sensor data, the set of focal distances between the camera and objects-of-interest in the field-of-view comprises:
generating a depth map defining the set of focal distances between the camera and the objects-of-interest in the field-of-view of the camera.
7. The method of claim 4 , wherein inferring a plurality of segments in the field-of-view is performed utilizing the sensor data.
8. The method of claim 1 , wherein combining the at least one of the sample images captured at the one or more focal distances swept by the autofocus lens with the other image to produce the all-in-focus image further comprises:
layering the at least one of the sample images captured at the one or more focal distances swept by the autofocus lens with the other image; and
adjusting an alpha-channel of each of the sample images to control transparency or opacity and sharpen areas or objects-of-interest at each of the set of focal distances.
9. The method of claim 1 , wherein combining the at least one of the sample images captured at the one or more focal distances swept by the autofocus lens with the other image to produce the all-in-focus image further comprises:
extracting a portion of the at least one of the sample images captured at the one or more focal distances swept by the autofocus lens;
extracting a portion of the other image; and
arranging the portion of the at least one of the sample images captured at the one or more focal distances swept by the autofocus lens adjacent to the portion of the other image in producing the all-in-focus image.
10. The method of claim 1 , wherein sweeping the autofocus lens of the camera to the one or more of the focal distances from the set of focal distances comprises:
prior to driving the autofocus lens to a second focal distance of the one or more of the focal distances, driving the autofocus lens to a second focal distance of the one or more of the focal distances for a sufficient time to capture the sample image at the second focal distance.
11. A method comprising:
capturing a plurality of buffer images of a field-of-view of a camera;
storing the buffer images at an image buffer;
generating a depth map, the depth map defining focal distances between the camera and objects-of-interest in the field-of-view;
segmenting the depth map into a set of focal distances, each focal distance of the set of focal distances corresponding to a different segment of a plurality of segments, each segment of the plurality of segments corresponding to a focal distance of the set of focal distances and defining a focus area within the field-of-view of the camera, the focus area corresponding with objects-of-interest at similar focal distances from the camera;
determining whether a buffer image of the buffer images captures the field-of-view of the camera at any of the focal distances in the set of focal distances;
responsive to determining that a buffer image captures the field-of-view of the camera at a first focal distance in the set of focal distances, sweeping an autofocus lens of the camera to one or more of the focal distances from the set of focal distances other than the first focal distance;
capturing sample images at one or more of the focal distances from the set of focal distances swept by the autofocus lens;
combining at least one of the sample images captured at the one or more focal distances swept by the autofocus lens with another image to produce an all-in-focus image; and
outputting, for display, an indication of the all-in-focus image.
12. The method of claim 11 , wherein combining at least one of the sample images captured at the one or more focal distances swept by the autofocus lens with the other image to produce the all-in-focus image further comprises:
combining one of the sample images with one of the buffer images.
13. The method of claim 11 , wherein combining at least one of the sample images captured at the one or more focal distances swept by the autofocus lens with the other image to produce the all-in-focus image further comprises:
combining a second sample image with the other image to produce the all-in-focus image, wherein the second sample image is different than the at least one of the sample images being combined to produce the all-in-focus image.
14. The method of claim 11 , further comprising:
obtaining sensor data, from at least one of a depth sensor, a contrast sensor, or a phase-detection-sensor; and
at least one of:
determining a set of focal distances between the camera and objects-of-interest in the field-of-view is performed utilizing the sensor data; or
inferring a plurality of segments in the field-of-view is performed utilizing the sensor data.
15. The method of claim 14 , wherein determining, based on sensor data, the set of focal distances between the camera and objects-of-interest in the field-of-view comprises:
generating a depth map defining the set of focal distances between the camera and the objects-of-interest in the field-of-view of the camera.
16. A method comprising:
generating a depth map of a field-of-view of a camera, the depth map defining focal distances between a camera and objects-of-interest in the field-of-view of the camera;
based on at least one buffer image of the field-of-view of the camera, sweeping an autofocus lens of the camera to one or more of the focal distances defined by the depth map;
capturing a sample image at each of the focal distances swept by the autofocus lens, each captured sample image comprising an in-focus portion at the focal distance swept by the autofocus lens;
producing an all-in-focus image by combining together the in-focus portions of the captured sample images with at least one portion of the at least one buffer image of the field-of-view of the camera; and
outputting, for display, an indication of the all-in-focus image produced.
17. The method of claim 16 , further comprising:
obtaining sensor data, from at least one of a depth sensor, a contrast sensor, or a phase-detection-sensor; and
at least one of:
determining a set of focal distances between the camera and objects-of-interest in the field-of-view is performed utilizing the sensor data; or
inferring a plurality of segments in the field-of-view is performed utilizing the sensor data.
18. The method of claim 17 , wherein determining, based on sensor data, the set of focal distances between the camera and objects-of-interest in the field-of-view comprises:
generating a depth map defining the set of focal distances between the camera and the objects-of-interest in the field-of-view of the camera.