IP Library › Granted Patent US 12,445,705
Granted Patent B2
US 12,445,705 · App. 17/890,982 · Granted Oct 14, 2025

Using guided filter to enhance depth estimation with brightness image

Inventors: Kevin Joseph Mcandrew (Galway, IE); Krystian Balicki (Wroclaw, PL); Charles Mathy (Arlington, MA); Nicolas Le Dortz (Jamaica Plain, MA); Jonathan Ephraim David Hurwitz (Edinburgh, GB); Eoin E. English (Pallasgreen, IE); Javier Calpe Maravilla (Valencia, ES)
Assignee: Analog Devices International Unlimited Company
H04N23/56G01S7/4915G01S17/36G01S17/894G06T5/00G06T5/20G06T5/50G06T7/13G06T7/521G06T7/55H04N23/11H04N23/74H04N25/705H04N25/78G06T2207/10028G06T2207/10048G06T2207/20192G06T2207/20221
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,445,705
App. No.
17/890,982
Granted
Oct 14, 2025
Kind
B2
Abstract

A depth estimation system can use a guided filter to enhance depth estimation using brightness image. Light is projected onto an object. The object reflects at least a portion of the projected light. The reflected light is at least partially captured by an image sensor. The depth estimation system may generate a depth image based on a phase shift between the captured light and the projected light and generate a brightness image based on brightness of the captured light. The depth estimation system may use the guided filter to identify a pixel that represents at least a portion of a boundary of the object. The guided filter can determine a depth value of the pixel based on the value of the corresponding pixel in the brightness image. The depth estimation system can assign the depth value to the pixel and generates an enhanced depth image.

Claims (53)

1. A method, comprising:

determining a weight for a target pixel of a depth image representing an object based on the depth image and a brightness image representing the object;

determining whether the target pixel represents at least a portion of a boundary of the object based on the weight;

in response to determining that the target pixel represents at least the portion of the boundary, determining a depth value of the target pixel based on a brightness value of a corresponding pixel in the brightness image; and

updating the depth image by assigning the depth value to the target pixel.

2. The method of claim 1 , further comprising:

in response to determining that the target pixel does not represent any portion of the boundary, determining the depth value for the target pixel based on depth values of other pixels in the depth image, wherein the other pixels surround the target pixel.

3. The method of claim 2 , wherein the depth value for the target pixel is a weighted average of the depth values of the other pixels.

4. The method of claim 1 , wherein the depth image is used as an input image of a guided filter, and the brightness image is used as a guidance image of the guided filter.

5. The method of claim 1 , wherein the depth image and the brightness image are generated based on image data from a same image sensor.

6. The method of claim 1 , further comprising:

instructing an illuminator assembly to project modulated light into a local area including the object;

instructing a camera assembly to capture reflected light from at least a portion of the object; and

generating the depth image based on a phase shift between the reflected light and the modulated light projected into the local area.

7. The method of claim 6 , further comprising:

generating the brightness image based on brightness of the reflected light.

8. The method of claim 6 , wherein the modulated light includes continuous waves of infrared light, the continuous waves having different modulation frequencies.

9. The method of claim 1 , wherein determining whether the target pixel represents at least the portion of the boundary of the object comprises:

determining whether a value of the weight is equal or substantially similar to 1; and

in response to determining that the value of the weight is equal or substantially similar to 1, determining that the target pixel represents at least the portion of the boundary of the object.

10. The method of claim 1 , wherein determining whether the target pixel represents at least the portion of the boundary of the object comprises:

determining whether a value of the weight is equal or substantially similar to 0; and

in response to determining that the value of the weight is equal or substantially similar to 0, determining that the target pixel does not represent any portion of the boundary of the object.

11. A system, comprising:

a camera assembly configured to capture reflected light from at least a portion of an object; and

a controller configured to:

generate a depth image based on the reflected light,

generate a brightness image based on the reflected light,

determine a weight for a target pixel of the depth image by applying a guided filter on the depth image and the brightness image,

determine whether the target pixel represents at least a portion of a boundary of the object based on the weight,

in response to determining that the target pixel represents at least the portion of the boundary, determine a depth value of the target pixel based on a brightness value of a corresponding pixel in the brightness image, and

generate an enhanced depth image based on the depth value of the target pixel.

12. The system of claim 11 , further comprising a display configured to present one or more images generated based on the enhanced depth image.

13. The system of claim 11 , wherein the controller is further configured to generate a navigation instruction for a mobile device based on the enhanced depth image.

14. The system of claim 11 , wherein the controller is further configured to:

in response to determining that the target pixel does not represent any portion of the boundary, determine the depth value for the target pixel based on depth values of other pixels in the depth image, wherein the other pixels surround the target pixel.

15. The system of claim 11 , wherein the controller is further configured to generate the depth image and the brightness image based on the reflected light by:

generating the depth image based on a phase shift between the reflected light and modulated light projected into a local area including the object; and

generating the brightness image based on brightness of the reflected light.

16. One or more non-transitory computer-readable media storing instructions executable to perform operations, the operations comprising:

determining a weight for a target pixel of a depth image capturing an object based on the depth image and a brightness image capturing the object;

determining whether the target pixel represents at least a portion of a boundary of the object based on the weight;

in response to determining that the target pixel represents at least the portion of the boundary, determining a depth value of the target pixel based on a brightness value of a corresponding pixel in the brightness image; and

updating the depth image by assigning the depth value to the target pixel.

17. The one or more non-transitory computer-readable media of claim 16 , wherein the operations further comprise:

in response to determining that the target pixel does not represent any portion of the boundary, determining the depth value for the target pixel based on depth values of other pixels in the depth image, wherein the other pixels surround the target pixel.

18. The one or more non-transitory computer-readable media of claim 16 , wherein the depth image is used as an input image of a guided filter, and the brightness image is used as a guidance image of the guided filter.

19. The one or more non-transitory computer-readable media of claim 16 , wherein the depth image and the brightness image are generated based on image data from a same image sensor.

20. The one or more non-transitory computer-readable media of claim 16 , wherein the operations further comprise:

instructing an illuminator assembly to project modulated light into a local area including the object;

instructing a camera assembly to capture reflected light from at least a portion of the object;

generating the depth image based on a phase shift between the reflected light and the modulated light projected into the local area; and

generating the brightness image based on brightness of the reflected light.

Assignments (2)
CORRECTIVE ASSIGNMENT TO CORRECT THE SIXTH INVENTOR'S NAME PREVIOUSLY RECORDED ON REEL 60849 FRAME 439. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT OF ASSIGNOR'S INTEREST.. Recorded Sep 19, 2025
From: MCANDREW, KEVIN JOSEPH; BALICKI, KRYSTIAN; MATHY, CHARLES; LE DORTZ, NICOLAS; HURWITZ, JONATHAN EPHRAIM DAVID; ENGLISH, EOIN E.; CALPE MARAVILLA, JAVIER
To: ANALOG DEVICES INTERNATIONAL UNLIMITED COMPANY
Reel/Frame 072910/0106 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 18, 2022
From: MCANDREW, KEVIN JOSEPH; BALICKI, KRYSTIAN; MATHY, CHARLES; LE DORTZ, NICOLAS; HURWITZ, JONATHAN EPHRAIM DAVID; ENGLISH, EOIN; CALPE MARAVILLA, JAVIER
To: ANALOG DEVICES INTERNATIONAL UNLIMITED COMPANY
Reel/Frame 060849/0439 →
Continuity (2)
Provisional Application 63310859 · Feb 16, 2022
Related Publication 20230260094A1 · Aug 17, 2023
References Cited (45)
US 8619122B2 · Gilboa · 2013 [cited by examiner]
US 8660337B2 · Takahashi · 2014 [cited by examiner]
US 8687044B2 · Katz · 2014 [cited by examiner]
US 8717414B2 · Kang · 2014 [cited by examiner]
US 8885920B2 · Lim · 2014 [cited by examiner]
US 8983233B2 · Katz · 2015 [cited by examiner]
US 9167230B2 · Min et al. · 2015 [cited by applicant]
US 10091491B2 · Choi · 2018 [cited by examiner]
US 10132616B2 · Wang · 2018 [cited by applicant]
US 10535151B2 · Bleyer et al. · 2020 [cited by applicant]
US 10931905B2 · Jin et al. · 2021 [cited by applicant]
US 11204415B2 · Jin et al. · 2021 [cited by applicant]
US 11734801B2 · Ortiz Egea · 2023 [cited by applicant]
US 20110285910A1 · Bamji et al. · 2011 [cited by applicant]
US 20130176426A1 · Ovsiannikov et al. · 2013 [cited by applicant]
US 20160330434A1 · Chen · 2016 [cited by examiner]
US 20170018114A1 · Stewart et al. · 2017 [cited by applicant]
US 20180343438A1 · Cho et al. · 2018 [cited by applicant]
US 20190197735A1 · Xiong et al. · 2019 [cited by applicant]
US 20210356598A1 · Hurwitz · 2021 [cited by applicant]
Ferstl et al., “Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation,” Proceedings of the IEEE International Conference on Computer Vision (ICCV), IEEE, Dec. 1, 2013, pp. 993-1000. [cited by applicant]
Hui et al., “Depth Map Super-Resolution by Deep Multi-Scale Guidance,” 16th European Conference—Computer Vision, Sep. 17, 2016, pp. 353-369. [cited by applicant]
International Search Report and Written Opinion in PCT/EP2023/053961, mailed May 15, 2023, 21 pages. [cited by applicant]
International Search Report and Written Opinion in PCT/EP2023/053974, mailed May 15, 2023, 19 pages. [cited by applicant]
International Search Report and Written Opinion in PCT/EP2023/053978, mailed May 15, 2023, 20 pages. [cited by applicant]
Jung et al., “Intensity-guided edge-preserving depth upsampling through weighted L0 gradient minimization,” Journal of Visual Communication and Image Representation, Nov. 17, 2016, vol. 42, pp. 132-144. [cited by applicant]
Li et al., “Guided Depth Map Super-Resolution Using Recumbent Y Network,” IEEE Access, IEEE, Jul. 7, 2020, vol. 8, pp. 122695-122708. [cited by applicant]
Or-El et al., “RGBD-fusion: Real-time high precision depth recovery,” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jun. 7, 2015, pp. 5407-5416. [cited by applicant]
Chen et al., Depth image enhancement for kinect using region growing and bilateral filter, in Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012). IEEE, 2012, pp. 3070-3073. [cited by applicant]
Evangelidis et al., Fusion of range and stereo data for high-resolution scene-modeling, IEEE transactions on pattern analysis and machine intelligence, 2015, vol. 37, No. 11, pp. 2178-2192. [cited by applicant]
Gu et al., Learning dynamic guidance for depth image enhancement, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3769-3778. [cited by applicant]
Izadi et al., Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera, in Proceedings of the 24th annual ACM symposium on User interface software and technology, 2011, pp. 559-568. [cited by applicant]
Jeon et al., Reconstruction-based pairwise depth dataset for depth image enhancement using cnn, in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 422-438. [cited by applicant]
Kim et al., Spatial and temporal enhancement of depth images captured by a time-of-flight depth sensor, in 2010 20th International Conference on Pattern Recognition. IEEE, 2010, pp. 2358-2361. [cited by applicant]
Mutto et al., Locally consistent tof and stereo data fusion, in European Conference on Computer Vision. Springer, 2012, pp. 598-607. [cited by applicant]
Park et al., High quality depth map upsampling for 3d-tof cameras, in 2011 International Conference on Computer Vision. IEEE, 2011, pp. 1623-1630. [cited by applicant]
Russakoff et al., Image similarity using mutual information of regions, in European Conference on Computer Vision. Springer, 2004, pp. 596-607. [cited by applicant]
Schwarz et al., A weighted optimization approach to time-of-flight sensor fusion, IEEE Transactions on Image Processing, 2013, vol. 23, No. 1, pp. 214-225. [cited by applicant]
Schwarz et al., Improved edge detection for ewoc depth upscaling, in 2012 19th International Conference on Systems, Signals and Image Processing (IWSSIP). IEEE, 2012, pp. 1-4. [cited by applicant]
Schwarz et al., Time-of-flight sensor fusion with depth measurement reliability weighting, in 2014 3DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON). IEEE, 2014, pp. 1-4. [cited by applicant]
Sublime et al., Un algorithme ICM basé sur la compacité pour la segmentation des images satellites à très haute résolution, EGC, 2015, pp. 191-196. [cited by applicant]
Zakeri et al., Guided optimization framework for the fusion of time-of-flight with stereo depth, Journal of Electronic Imaging, 2020, vol. 29, No. 5, pp. 053016. [cited by applicant]
Zhang, Image restoration: flexible neighborhood systems and iterated conditional expectations, Statistica Sinica, 1993, pp. 117-139. [cited by applicant]
Zhu et al., Reliability fusion of time-of-flight depth and stereo geometry for high quality depth maps, IEEE transactions on pattern analysis and machine intelligence, 2010, vol. 33, No. 7, pp. 1400-1414. [cited by applicant]
Zivkovic et al., Gentle icm energy minimization for markov random fields with smoothness-based priors, Journal of Real-Time Image Processing, 2016, vol. 11, No. 1, pp. 235-246. [cited by applicant]