IP Library Granted Patent US 12,192,652
Granted Patent B2
US 12,192,652 · App. 18/482,117 · Granted Jan 7, 2025

Method and apparatus for estimating a value in a table generated by a photosites matrix

Inventors: Valentin Rebiere (Grenoble, FR); Antoine Drouot (Paris, FR)
Assignee: STMicroelectronics France
H04N25/60H04N23/843H04N25/13H04N25/705
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,192,652
App. No.
18/482,117
Granted
Jan 7, 2025
Kind
B2
Abstract

An embodiment method for estimating a missing or incorrect value in a table of values generated by a photosite matrix comprises a definition of a zone of the table comprising the value to be estimated and other values, referred to as neighboring values, and an estimation of the value to be estimated based on the primary neighboring values and the weight associated with these primary neighboring values, wherein a weight of each neighboring value, referred to as primary neighboring value, of the same colorimetric component as that of the missing or incorrect value to be estimated, is determined according to differences between neighboring values disposed on an axis and neighboring values disposed parallel with this axis and positioned in relation to this axis on the same side as the primary neighboring value for which the weight is determined.

Claims (39)

1. A method for estimating a missing or incorrect value, the method comprising:

defining a kernel of a table of values comprising a value to be estimated and neighboring values, the table of values being generated by a photosite matrix, the kernel having a missing or incorrect value location surrounded by locations of values generated by photosites, the missing or incorrect value location being the value to be estimated;

determining a weight for each primary neighboring value of a same colorimetric component as that of the missing or incorrect value to be estimated;

calculating semi-gradients, based on differences between neighboring values of the table, each semi-gradient being associated with a given direction with respect to the location of the missing or incorrect value to be determined, the weight of each primary neighboring value being determined based on the semi-gradient associated with a direction defined between the location of each primary neighboring value and the location of the missing or incorrect value; and

estimating the value to be estimated based on the primary neighboring values and the weight associated with the primary neighboring values.

2. The method of claim 1 , wherein the photosite matrix is a Bayer matrix.

3. The method of claim 1 , wherein the kernel has a size of 5×5 value locations.

4. The method of claim 1 , wherein eight semi-gradients are calculated for eight given directions along four axes passing through the location of the missing or incorrect value.

5. The method of claim 4 , wherein the four axes are one horizontal axis, one vertical axis, and two diagonal axes.

6. The method of claim 1 , wherein estimating the value comprises a discrimination of the semi-gradients by a Gaussian function, the weights of the primary neighboring values being calculated based on the discriminated semi-gradients.

7. The method of claim 1 , wherein estimating the value comprises a discrimination of the semi-gradients by a linear approximation of a Gaussian function.

8. The method of claim 1 , wherein the kernel is centered on the missing or incorrect value.

9. The method of claim 1 , wherein each semi-gradient is obtained from a sum of differences between neighboring values located on an axis passing through the missing or incorrect value and orthogonal to an orientation of the semi-gradient and neighboring values located in locations extending parallel with this axis and delimiting the kernel in the direction of the semi-gradient.

10. A computer program product stored on a storage device, the computer program product comprising instructions to be executed by a microprocessor, the instructions when executed causing the microprocessor to:

define a kernel of a table of values comprising a value to be estimated and neighboring values, the table of values being generated by a photosite matrix, the kernel having a missing or incorrect value location surrounded by locations of values generated by photosites, the missing or incorrect value location being the value to be estimated;

determine a weight for each primary neighboring value of a same colorimetric component as that of the missing or incorrect value to be estimated;

define, in the table, several axes passing through a location of the missing or incorrect value to be estimated, the weight of each primary neighboring value being determined according to differences between neighboring values disposed in the table on one of the defined axes and neighboring values disposed in the table parallel with the one of the defined axes and positioned in relation to the one of the defined axes on a same side as the primary neighboring value for which the weight is determined;

calculate semi-gradients, based on the differences, each semi-gradient being associated with a given direction with respect to the location of the missing or incorrect value to be determined, the weight of each primary neighboring value being determined based on the semi-gradient associated with a direction defined between the location of each primary neighboring value and the location of the missing or incorrect value; and

estimate the value to be estimated based on the primary neighboring values and the weight associated with the primary neighboring values.

11. The computer program product of claim 10 , wherein the kernel is centered on the missing or incorrect value to be estimated.

12. The computer program product of claim 10 , wherein the kernel has a size of 5×5 value locations.

13. The computer program product of claim 10 , wherein the instructions are further configured to calculate at least two main gradients along two orthogonal orientations in relation to one another, each main gradient being calculated based on differences between the neighboring values disposed on one side of an axis passing through the location of the missing or incorrect value to be estimated and the neighboring values disposed on an other side of the axis, the axis being orthogonal to the orientation of the each main gradient.

14. The computer program product of claim 13 , wherein the instructions are further configured to compare between two mutually orthogonal main gradients so as to detect a dominant colorimetric rupture line passing through the location of the missing value.

15. The computer program product of claim 10 , wherein the photosites of the photosite matrix are disposed according to a Bayer matrix.

16. An apparatus comprising:

a photosite matrix comprising photosites; and

a data acquisition and processing unit comprising a microprocessor, the data acquisition and processing unit being configured to:

acquire a table of values proportional to an illumination of the photosites,

wherein the table comprises a missing value at a location of one depth sensor, and wherein the missing value and each neighboring value of the table are associated with a colorimetric component;

define a kernel of the table comprising the missing value and the neighboring values, the location of the missing value being surrounded by locations of neighboring values;

determine a weight for each primary neighboring value of a same colorimetric component as that of the missing value;

define, in the table, several axes passing through a location of the missing value, wherein the weight of each primary neighboring value is determined according to differences between neighboring values disposed in the table on one of the defined axes and neighboring values disposed in the table parallel with the one of the defined axes and positioned in relation to the one of the defined axes on a same side as the primary neighboring value for which the weight is determined;

calculate semi-gradients, based on the differences, each semi-gradient being associated with a given direction with respect to the location of the missing value, the weight of each primary neighboring value being determined based on the semi-gradient associated with a direction defined between the location of each primary neighboring value and the location of the missing value;

estimate the missing value based on the primary neighboring values and the weight associated with the primary neighboring values; and

correct the missing value.

17. The apparatus of claim 16 , wherein the apparatus is a mobile telephone.

18. The apparatus of claim 16 , wherein the apparatus is a tablet.

19. The apparatus of claim 16 , wherein each photosite is associated with red, green, or blue.

20. The apparatus of claim 16 , wherein the data acquisition and processing unit comprises a non-programmable integrated circuit.

Assignments (2)
CHANGE OF NAME Recorded Dec 8, 2023
From: STMICROELECTRONICS SA
To: STMICROELECTRONICS FRANCE
Reel/Frame 065835/0159 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 6, 2023
From: REBIERE, VALENTIN; DROUOT, ANTOINE
To: STMICROELECTRONICS SA
Reel/Frame 065144/0817 →
Priority Claims (1)
FR 2002673 · Mar 18, 2020 · national
Continuity (2)
Continuation 17199444 · Mar 12, 2021
Related Publication 20240048863A1 · Feb 8, 2024
References Cited (25)
US 3971065A · Bayer · 1976 [cited by applicant]
US 8873847B2 · Alacoque · 2014 [cited by applicant]
US 20090066821A1 · Achong · 2009 [cited by examiner]
US 20130051665A1 · Shinozaki · 2013 [cited by examiner]
US 20140368701A1 · Shi · 2014 [cited by examiner]
US 20150022869A1 · Shi · 2015 [cited by examiner]
US 20180288281A1 · Rangnekar · 2018 [cited by examiner]
US 20190379336A1 · Takeuchi · 2019 [cited by examiner]
Alleysson, D., et al., “Linear Demosaicing Inspired by the Human Visual System”, IEEE Trans. Image Process., vol. 14, No. 4, Apr. 2005, 12 pages. [cited by applicant]
Chang, L., et al., “Hybrid Color Filter Array Demosaicking for Effective Artifact Suppression”, J. Electron. Imaging, vol. 15, No. 1, Jan.-Mar. 2006, 40 pages. [cited by applicant]
Foix, S., et al., “Lock-in Time-of-Flight (ToF) Cameras: A Survey”, IEEE Sens. J., vol. 11, No. 3, Sep. 2011, 11 pages. [cited by applicant]
Fossum, E. R., et al., “A Review of the Pinned Photodiode for CCD and CMOS Image Sensors”, IEEE J. Electron Devices Soc., vol. 2, No. 3, May 2014, 34 pages. [cited by applicant]
Gunturk, B. K., et al., “Demosaicking: Color filter Array Interpolation”, IEEE Signal Process. Mag., (44), Jan. 2005, 11 pages. [cited by applicant]
Hore, A., et al., “An Edge-Sensing Generic Demosaicing Algorithm with Application to Image Resampling”, IEEE Trans. Image Process., vol. 20, No. 11, Nov. 2011, pp. 3136-3150. [cited by applicant]
Je, C., et al., “High-Contrast Color-Stripe Pattern for Rapid Structured-Light Range Imaging”, Lect. Notes Comput. Sci., vol. 3021, May 10, 2004, 13 pages. [cited by applicant]
Jin, S., et al., “Fine Directional De-interlacing Algorithm Using Modified Sobel Operation”, IEEE Trans. Consum. Electron., vol. 54, No. 2, May 2008, pp. 857-862. [cited by applicant]
Kim, W., et al., “A 1.5Mpixel RGBZ CMOS Image Sensor for Simultaneous Color and Range Image Capture,” Digest of Technical Papers, IEEE International Solid-State Circuits Conference, vol. 55, No. Jun. 2011, pp. 392-393. [cited by applicant]
Kimmel, R., “Demosaicing: Image Reconstruction from Color CCD Samples”, IEEE Transactions on Image Processing, vol. 8, No. 9, Sep. 1999, 8 pages. [cited by applicant]
Kodak “Kodak Lossless True Color Image Suite”, http://r0k.us/graphics/kodak/, Nov. 15, 1999, 4 pages. [cited by applicant]
Microsoft, “Xbox 360 Kinect Sensor Manual”, Nov. 2010, 19 pages. [cited by applicant]
Pei, S., et al., “Effective Color Interpolation in CCD Color Filter Arrays Using Signal Correlation”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, No. 6, Jun. 2003, pp. 503-513. [cited by applicant]
Shi, L., et al., “Demosaicing for RGBZ Sensor, ”Proceedings of SPIE-IS&T Electronic Imaging, vol. 8657, 2013, 9 pages, Downloaded From: http://proceedings.spiedigitallibrary.org/ on Jul. 12, 2016 Terms of Use: http://sp… [cited by applicant]
Wang, Z. et al., “Image Quality Assessment: From Error Visibility to Structural Similarity”, IEEE Transactions Image Processing, vol. 13, No. 4, Apr. 2004, 14 pages. [cited by applicant]
Xiangxi, Z., et al., “FPGA implementation of edge detection for Sobel operator in eight directions”, IEEE Asia Pacific Conference on Circuits and Systems, APCCAS, Oct. 26-30, 2018, pp. 520-523. [cited by applicant]
Zhang, L., et al., “Color demosaicking by local Directional interpolation and nonlocal adaptive thresholding”, Journal of Electronic Imaging, 20(2), 023016, Apr.-Jun. 2011, pp. 1-29. [cited by applicant]