IP Library › Granted Patent US 12,333,685
Granted Patent B2
US 12,333,685 · App. 18/506,812 · Granted Jun 17, 2025

Automatic generation of all-in-focus images with a mobile camera

Inventors: Szepo Robert Hung (Austin, TX); Ying Chen Lou (Santa Clara, CA)
Assignee: Google LLC
G06T5/50G03B13/36G06T5/73H04N23/676H04N23/80H04N23/959G06T2207/10148G06T2207/20221
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,333,685
App. No.
18/506,812
Granted
Jun 17, 2025
Kind
B2
Abstract

The present disclosure describes systems and techniques directed to producing an all-in-focus image with a camera of a mobile device, in particular, cameras with shallow depth-of-field. User equipment includes a sensor for determining distance to an object in a camera's field-of-view. Based on a depth map of the field-of-view, a plurality of segments is inferred, each segment defining a unique focus area within the camera's field-of-view. An autofocus lens of the camera sweeps to a respective focal distance associated with each of the plurality of segments. The camera captures sample images at each focal distance swept by the autofocus lens. The user equipment produces an all-in-focus image by combining or merging portions of the captured sample images.

Claims (46)

1. A method comprising:

generating a depth map of a field-of-view of a camera, the depth map defining focal distances between a camera and objects-of-interest in the field-of-view of the camera;

sweeping an autofocus lens of the camera to one or more of the focal distances defined by the depth map;

capturing a sample image at each of the focal distances swept by the autofocus lens, each captured sample image comprising an in-focus portion at the focal distance swept by the autofocus lens; and

producing an all-in-focus image by combining together the in-focus portions of each captured sample image with at least one portion of at least one buffer image of the field-of-view of the camera, wherein the at least one buffer image is at least one previous image of the field-of-view of the camera captured prior to sweeping the autofocus lens, and wherein sweeping the autofocus lens of the camera to the one or more focal distances comprises refraining from sweeping the autofocus lens to at least one focal distance associated with the at least one buffer image.

2. The method of claim 1 , wherein the depth map is based on sensor data from a depth sensor, a contrast sensor, or a phase-detection sensor.

3. The method of claim 1 , wherein the autofocus lens comprises a voice coil motor lens.

4. The method of claim 1 , wherein combining together the in-focus portions of each captured sample image with the at least one portion of the at least one buffer image of the field-of-view of the camera comprises:

layering the sample image captured at each of the focal distances with the at least one portion of the at least one buffer image; and

adjusting an alpha-channel of each sample image to control transparency or opacity and sharpen areas or objects-of-interest at each of the one or more focal distances.

5. The method of claim 1 , wherein combining together the in-focus portions of each captured sample image with the at least one portion of the at least one buffer image of the field-of-view of the camera comprises:

extracting a portion of the sample image captured at each of the focal distances; and

arranging the portion of the sample image captured at each of the focal distances adjacent to the at least one portion of at least one buffer image of the field-of-view of the camera in producing the all-in-focus image.

6. The method of claim 1 , further comprising:

automatically operating the camera in an all-in-focus mode based on the depth map.

7. The method of claim 6 , wherein automatically operating the camera in an all-in-focus mode based on the depth map comprises:

determining that the depth map includes two or more segments with respective focal distances that are at least a threshold distance apart; and

automatically operating the camera in the all-in-focus mode based on determining that the depth map includes the two or more segments with respective focal distances that are at least the threshold distance apart.

8. The method of claim 1 , wherein sweeping the autofocus lens of the camera to the one or more of the focal distances defined by the depth map comprises:

prior to driving the autofocus lens to a second focal distance of the one or more of the focal distances, driving the autofocus lens to a first focal distance of the one or more of the focal distances for a sufficient time to capture the sample image at the first focal distance.

9. The method of claim 8 , wherein the first focal distance is a nearest current focal distance of the autofocus lens prior to sweeping the autofocus lens of the camera.

10. The method of claim 1 , wherein sweeping the autofocus lens of the camera to the one or more of the focal distances defined by the depth map comprises driving the autofocus lens to each of the one or more of the focal distances in an order determined to minimize time that the autofocus lens is sweeping.

11. The method of claim 1 , wherein the sweeping of the autofocus lens of the camera to the one or more of the focal distances defined by the depth map is performed based on the at least one buffer image of the field-of-view of the camera.

12. A system comprising at least one processor configured to:

generate a depth map of a field-of-view of a camera, the depth map defining focal distances between a camera and objects-of-interest in the field-of-view of the camera;

sweep an autofocus lens of the camera to one or more of the focal distances defined by the depth map;

capture a sample image at each of the focal distances swept by the autofocus lens, each captured sample image comprising an in-focus portion at the focal distance swept by the autofocus lens; and

produce an all-in-focus image by combining together the in-focus portions of each captured sample image with at least one portion of at least one buffer image of the field-of-view of the camera, wherein the at least one buffer image is at least one previous image of the field-of-view of the camera captured prior to sweeping the autofocus lens, and wherein the at least one processor is configured to sweep the autofocus lens of the camera to the one or more focal distances by refraining from sweeping the autofocus lens to at least one focal distance associated with the at least one buffer image.

13. The system of claim 12 , wherein the depth map is based on sensor data from a depth sensor, a contrast sensor, or a phase-detection sensor.

14. The system of claim 12 , wherein the autofocus lens comprises a voice coil motor lens.

15. The system of claim 12 , wherein combining together the in-focus portions of each captured sample image with the at least one portion of the at least one buffer image of the field-of-view of the camera comprises:

layering the sample image captured at each of the focal distances with the at least one portion of the at least one buffer image; and

adjusting an alpha-channel of each sample image to control transparency or opacity and sharpen areas or objects-of-interest at each of the one or more focal distances.

16. The system of claim 12 , wherein combining together the in-focus portions of each captured sample image with the at least one portion of the at least one buffer image of the field-of-view of the camera comprises:

extracting a portion of the sample image captured at each of the focal distances; and

arranging the portion of the sample image captured at each of the focal distances adjacent to the at least one portion of at least one buffer image of the field-of-view of the camera in producing the all-in-focus image.

17. A non-transitory computer readable medium comprising program instructions executable by at least one processor to perform operations comprising:

generating a depth map of a field-of-view of a camera, the depth map defining focal distances between a camera and objects-of-interest in the field-of-view of the camera;

sweeping an autofocus lens of the camera to one or more of the focal distances defined by the depth map;

capturing a sample image at each of the focal distances swept by the autofocus lens, each captured sample image comprising an in-focus portion at the focal distance swept by the autofocus lens; and

producing an all-in-focus image by combining together the in-focus portions of each captured sample image with at least one portion of at least one buffer image of the field-of-view of the camera, wherein the at least one buffer image is at least one previous image of the field-of-view of the camera captured prior to sweeping the autofocus lens, and wherein sweeping the autofocus lens of the camera to the one or more focal distances comprises refraining from sweeping the autofocus lens to at least one focal distance associated with the at least one buffer image.

18. The non-transitory computer readable medium of claim 17 , wherein the depth map is based on sensor data from a depth sensor, a contrast sensor, or a phase-detection sensor.

19. The non-transitory computer readable medium of claim 17 , wherein the autofocus lens comprises a voice coil motor lens.

20. The non-transitory computer readable medium of claim 17 , wherein combining together the in-focus portions of each captured sample image with the at least one portion of the at least one buffer image of the field-of-view of the camera comprises:

layering the sample image captured at each of the focal distances with the at least one portion of the at least one buffer image; and

adjusting an alpha-channel of each sample image to control transparency or opacity and sharpen areas or objects-of-interest at each of the one or more focal distances.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 16, 2023
From: HUNG, SZEPO ROBERT; LOU, YING CHEN
To: GOOGLE LLC
Reel/Frame 065589/0617 →
Continuity (3)
Continuation 17628871
Continuation 16589025 · Sep 30, 2019
Related Publication 20240078639A1 · Mar 7, 2024
References Cited (86)
US 6320979B1 · Melen · 2001 [cited by applicant]
US 7099056B1 · Kindt · 2006 [cited by applicant]
US 8483479B2 · Kunkel et al. · 2013 [cited by applicant]
US 8798378B1 · Babenko et al. · 2014 [cited by applicant]
US 8890975B2 · Baba et al. · 2014 [cited by applicant]
US 8934666B2 · Lüke et al. · 2015 [cited by applicant]
US 9044171B2 · Venkatraman et al. · 2015 [cited by applicant]
US 9076204B2 · Ogura et al. · 2015 [cited by applicant]
US 9253412B2 · Lee · 2016 [cited by applicant]
US 9686475B2 · Neglur · 2017 [cited by applicant]
US 9704250B1 · Shah et al. · 2017 [cited by applicant]
US 9774798B1 · Evans et al. · 2017 [cited by applicant]
US 10389936B2 · Kozub et al. · 2019 [cited by applicant]
US 10521952B2 · Ackerson et al. · 2019 [cited by applicant]
US 10984513B1 · Hung · 2021 [cited by examiner]
US 20030068100A1 · Covell et al. · 2003 [cited by applicant]
US 20050099494A1 · Deng et al. · 2005 [cited by applicant]
US 20060050409A1 · George et al. · 2006 [cited by applicant]
US 20090207266A1 · Yoda · 2009 [cited by applicant]
US 20090225199A1 · Ferren · 2009 [cited by applicant]
US 20090303343A1 · Drimbarean et al. · 2009 [cited by applicant]
US 20110090303A1 · Wu et al. · 2011 [cited by applicant]
US 20110261217A1 · Muukki et al. · 2011 [cited by applicant]
US 20110267499A1 · Wan et al. · 2011 [cited by applicant]
US 20120070097A1 · Adams, Jr. · 2012 [cited by applicant]
US 20120154579A1 · Hampapur et al. · 2012 [cited by applicant]
US 20130177203A1 · Koo et al. · 2013 [cited by applicant]
US 20130314558A1 · Ju et al. · 2013 [cited by applicant]
US 20140002606A1 · Blayvas et al. · 2014 [cited by applicant]
US 20140009639A1 · Lee · 2014 [cited by applicant]
US 20140105520A1 · Matsumoto · 2014 [cited by applicant]
US 20150104074A1 · Vondran, Jr. et al. · 2015 [cited by applicant]
US 20150156388A1 · Neglur · 2015 [cited by applicant]
US 20150279012A1 · Brown et al. · 2015 [cited by applicant]
US 20160112637A1 · Laroia et al. · 2016 [cited by applicant]
US 20160117798A1 · Lin et al. · 2016 [cited by applicant]
US 20160248968A1 · Baldwin · 2016 [cited by applicant]
US 20160248979A1 · Ben Israel et al. · 2016 [cited by applicant]
US 20160309065A1 · Karafin et al. · 2016 [cited by applicant]
US 20160337570A1 · Tan et al. · 2016 [cited by applicant]
US 20160360091A1 · Lindskog et al. · 2016 [cited by applicant]
US 20170053167A1 · Ren et al. · 2017 [cited by applicant]
US 20170116932A1 · Musgrave et al. · 2017 [cited by applicant]
US 20180077210A1 · Hannuksela et al. · 2018 [cited by applicant]
US 20190132520A1 · Gupta et al. · 2019 [cited by applicant]
US 20190171908A1 · Salavon · 2019 [cited by applicant]
US 20210329150A1 · Wang et al. · 2021 [cited by applicant]
CN 102984530 · 2013 [cited by applicant]
CN 103002218 · 2013 [cited by applicant]
CN 105474622 · 2016 [cited by applicant]
CN 106134176 · 2016 [cited by applicant]
CN 106412426 · 2017 [cited by applicant]
CN 107787463 · 2018 [cited by applicant]
WO 2016144454 · 2016 [cited by applicant]
WO 2018005073 · 2018 [cited by applicant]
WO 2018188535 · 2018 [cited by applicant]
WO 2019070299 · 2019 [cited by applicant]
WO 2021066894 · 2021 [cited by applicant]
Benke, Darrell, “Improving Smartphone Cameras with Color Sensor Technology,” OpenSystems Media, Nov. 10, 2016, 5 pages. [cited by applicant]
Boiarshinov, Dmitrii, “Improving Autofocus Speed in Macrophotography Applications,” Technical Disclosure Commons—https://www.dcommons.org/dpubs_series/5133, May 11, 2022, 9 pages. [cited by applicant]
Chang et al., “Low-light Image Restoration with Short-and-long-exposure Raw Pairs,” Jul. 1, 2020, 12 pages. [cited by applicant]
Chen et al., “Automatic Zoom Based on Image Saliency,” Technical Disclosure Commons, https://www.tdcommons.org/dpubs_series/4640, Oct. 7, 2021, 11 pages. [cited by applicant]
Chen, Yen-Cheng, “Enhancing Image Quality of Photographs Taken by Portable Devices by Matching Images to High Quality Reference Images Using Machine Learning and Camera Orientation and Other Image Metadata,” Technical D… [cited by applicant]
Chinese Patent Office, Office Action mailed on Oct. 24, 2022, issued in connection with Chinese Patent Application No. 202080019378.X, 21 pages. [cited by applicant]
Cho, Taeg Sang, “Motion Blur Removal from Photographs,” Thesis, Massachusetts Institute of Technology, Sep. 2010, 143 pages. [cited by applicant]
Fried et al., “Perspective-Aware Manipulation of Portrait Photos,” SIGGRAPH '16 Technical Paper, Jul. 24-28, 2016, Anaheim, CA, Jul. 2016, 10 pages. [cited by applicant]
Gao et al., “Camera Sensor Exposure Control During Camera Launch”, Nov. 24, 2020, 7 pages. [cited by applicant]
Gao et al., “Scene Metering and Exposure Control for Enhancing High Dynamic Range Imaging,” Technical Disclosure Commons; Retrieved from https://www.tdcommons.org/dpubs_series/3092, Apr. 1, 2020, 12 pages. [cited by applicant]
Gao et al., “Utilizing Spectral Sensor Data and Location Data to Determine the Lighting Conditions of a Scene,” Technical Disclosure Commons; Retrieved from https://www.tdcommons.org/dpubs_series/2734, Dec. 4, 2019, 8 p… [cited by applicant]
Hollister, Sean, “How to guarantee the iPhone 13 Pro's macro mode is on,” Retrieved at: https://www.theverge.com/227 45578/iphone-13-pro-macro-mode-how-to, Oct. 26, 2021, 11 pages. [cited by applicant]
Hong et al., “Method of Capturing a Video and a Set of Selected High-Quality Images During Camera Shutter Long-Dress,” Technical Disclosure Commons; Retrieved from https://www.tdcommons.org/dpubs_series/2757, Dec. 12, 2… [cited by applicant]
International Bureau, International Preliminary Report on Patentability mailed on Apr. 5, 2022, issued in connection with International Patent Application No. PCT/US2020/037434, 8 pages. [cited by applicant]
International Searching Authority, International Search Report and Written Opinion, mailed on Sep. 23, 2020, issued in connection with International Patent Application No. PCT/US2020/037434, 14 pages. [cited by applicant]
Jackson et al., “The Creative and Technical Differences Between Full Frame and S-35”, Accessed from: https://vmi.Iv/raining/useful-stuff/differences-belween-full-frame-and-s-35, Feb. 2020, 19 pages. [cited by applicant]
Jain et al., “On Detecting GANs and Retouching Based Synthetic Alterations,” Jan. 26, 2019, 7 pages. [cited by applicant]
Konstantinova et al., “Fingertip Proximity Sensor with Realtime Visual-based Calibration,” Oct. 2016, 6 pages. [cited by applicant]
Lombardi et al., “Adaptive User Interface for a Camera Aperture within an Active Display Area,” Technical Disclosure Commons; Retrieved from https://www.tdcommons.org/dpubs_series/2719, Nov. 25, 2019, 12 pages. [cited by applicant]
Moraldo, Horacio Hernan, “Virtual Camera Image Processing,” Technical Disclosure Commons; Retrieved from https://www.dcommons.org/dpubs_series/3072, Mar. 30, 2020, 10 pages. [cited by applicant]
Mustaniemi et al., “LSD—Joint Denoising and Deblurring of Short and Long Exposure Images with CNNs,” Sep. 1, 2020, 21 pages. [cited by applicant]
Portmann et al., “Detection of Automated Facial Beautification by a Camera Application by Comparing a Face to a Rearranged Face,” Technical Disclosure Commons, Retrieved from https://www.tdcommons.org/dpubs_series/2943,… [cited by applicant]
Shih et al., “Techniques for Deblurring Faces in Images by Utilizing Multi-Camera Fusion,” Technical Disclosure Commons- https://www.tdcommons.org/dpubs_series/4274, May 5, 2021, 9 pages. [cited by applicant]
Taivala et al., “Techniques and Apparatuses for Variable-Display Devices to Capture Screen-Filling Images with a Maximized Field of View,” Technical Disclosure Commons; Retrieved from https://www.tdcommons.org/dpubs_ser… [cited by applicant]
Yang et al., “Improved Object Detection in an Image by Correcting Regions with Distortion,” Technical Disclosure Commons; Retrieved from https://www.tdcommons.org/dpubs_series/3090, Apr. 1, 2020, 8 pages. [cited by applicant]
Yang et al., “Object Detection and Tracking with Post-Merge Motion Estimation,” Technical Disclosure Commons, https://www.tdcommons.org/dpubs_series_ 4353, Jun. 8, 2021, 11 pages. [cited by applicant]
Yang et al., “Using Image-Processing Settings to Determine an Optimal Operating Point for Object Detection on Imaging Devices,” Technical Disclosure Commons; Retrieved from https://www.tdcommons.org/dpubs_series/2985, M… [cited by applicant]
Yuan et al., “Image Deblurring with Blurred/Noisy Image Pairs,” Jul. 29, 2007, 10 pages. [cited by applicant]