IP Library › Granted Patent US 12,075,170
Granted Patent B2
US 12,075,170 · App. 17/819,586 · Granted Aug 27, 2024

Method of generating infrared image

Inventors: Soeren Smits (Neustadt, DE); Ulrich Pitz (Siefersheim, DE); Stepan Pazekha (Wuppertal, DE)
Assignee: Aptiv Technologies AG
H04N25/131H04N9/64H04N23/56H04N23/76
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,075,170
App. No.
17/819,586
Filed
Aug 12, 2022
Granted
Aug 27, 2024
Kind
B2
Art Unit
2638
USPC
348/332
Abstract

The method comprises a step of processing pixel values provided at the output of an image sensor comprising a two-dimensional array of pixels including a mix of IR pixels and color pixels. The step of processing includes, for each color pixel, determining a local color-related ratio indicative of a part of infrared light reception in the color pixel, and estimating an infrared portion value from a color pixel value based on the local color-related ratio. The method further includes generating an infrared image from pixel values from the IR pixels and the estimated infrared portion values from the color pixels.

Claims (64)

1. A method of generating an infrared image, comprising:

processing pixel values provided at an output of an image sensor comprising a two-dimensional array of pixels including a mix of IR pixels and color pixels, wherein processing includes, for each color pixel:

determining a local color-related ratio indicative of a part of infrared light reception in the color pixel; and

estimating an infrared portion value from a color pixel value based on the local color-related ratio;

generating an infrared image from pixel values from the IR pixels and the estimated infrared portion values from the color pixels;

prior to processing the pixel values, first preprocessing including adjusting an infrared transmittance of the color pixels in the array of pixels to a target level; and

second preprocessing performed after the first preprocessing, for each IR pixel, the second preprocessing including:

determining color portion values, representative of the part of color light reception in the IR pixel, in all color channels, from the pixel value of said IR pixel and predetermined color-related IR contamination gain coefficients; and

filtering the pixel value of said IR pixel by removing the determined color portion values.

2. The method according to claim 1 , wherein determining the local color-related ratio for a color pixel includes deriving said local color-related ratio from the pixel values of at least one IR pixel positioned adjacent to said color pixel in the array of pixels.

3. The method according to claim 2 , wherein determining the local color-related ratio for said color pixel includes deriving said local color-related ratio from the pixel values of one or more color pixels having the same color as said color pixel, and that are positioned adjacent to said at least one IR pixel in the array of pixels.

4. The method according to claim 1 , wherein determining said local color-related ratio for said color pixel includes:

determining a plurality of color-related ratios respectively related to a plurality of IR pixels positioned adjacent to said color pixel in the array of pixels; and

averaging said plurality of color-related ratios.

5. The method according to claim 1 , wherein estimating an infrared portion value for each color pixel includes applying the local color-related ratio to a pixel value of said color pixel.

6. The method according to claim 1 , wherein adjusting the infrared transmittance of the color pixels to the target level includes applying color-related normalization factors to the color pixel values.

7. The method according to claim 6 , wherein the color-related normalization factors are determined by:

illuminating with an only infrared light signal a target that is monotonically reflective in a light spectrum from visible spectrum to infrared spectrum;

receiving by the image sensor the infrared light signal reflected by said target;

averaging all IR pixel values to determine a mean IR value;

for each color channel, averaging all color pixel values to determine a mean color value; and

calculating each color-related normalization factor by calculating the ratio between the mean IR value and the mean color value.

8. The method according to claim 1 , wherein determining the color portion values for each IR pixel includes:

for each color channel:

determining a local color pixel value from at least one color pixel of said color channel positioned adjacent to said IR pixel; and

filtering said local color pixel value by removing an infrared light reception part value that is estimated as equal to the pixel value of said IR pixel; and

determining said color portion value of the IR pixel by applying a predetermined color-related IR contamination gain coefficient to said filtered local color pixel value.

9. The method according to claim 8 , wherein of determining, for each color channel, a local color pixel value includes averaging the color pixel values of a plurality of color pixels positioned adjacent to said IR pixel.

10. The method according to claim 1 , further comprising weighting said color portion values of each IR pixel by respective color-related intensity weights depending on detected light conditions in an environment of the image sensor.

11. The method according to claim 1 , further comprising:

determining the color-related IR contamination gain coefficient for each color channel by:

illuminating a target that is monotonically reflective in a light spectrum from visible spectrum to infrared spectrum, with a monochromatic light signal of said color;

receiving by the image sensor the monochromatic light signal reflected by said target;

averaging all IR pixel values to determine a first mean color value;

averaging all color pixel values related to said color to determine a second mean color value; and

calculating the color-related IR contamination gain coefficient by calculating the ratio between the first mean color value and the second mean color value.

12. An imaging system comprising:

an image sensor comprising a two-dimensional array of pixels including color pixels and IR pixels; and

an image processor for processing pixel values provided at an output of the image sensor, the image processor configured to:

process the pixel values for each color pixel by:

determining a local color-related ratio indicative of a part of infrared light reception in the color pixel; and

estimating an infrared portion value from a color pixel value based on the local color-related ratio; and

generate an infrared image from pixel values from the IR pixels and the estimated infrared portion values from the color pixels;

prior to processing the pixel values, perform a first preprocess including adjusting an infrared transmittance of the color pixels in the array of pixels to a target level;

perform a second preprocess, after the first preprocessing, for each IR pixel, the second preprocess including:

determining color portion values, representative of the part of color light reception in the IR pixel, in all color channels, from the pixel value of said IR pixel and predetermined color-related IR contamination gain coefficients; and

filtering the pixel value of said IR pixel by removing the determined color portion values.

13. The imaging system of claim 12 , wherein the image processor is further configured to determine the local color-related ratio for a color pixel by deriving said local color-related ratio from the pixel values of at least one IR pixel positioned adjacent to said color pixel in the array of pixels.

14. The imaging system of claim 13 , wherein the image processor is further configured to determining the local color-related ratio for said color pixel by deriving said local color-related ratio from the pixel values of one or more color pixels having the same color as said color pixel, and that are positioned adjacent to said at least one IR pixel in the array of pixels.

15. The imaging system of claim 13 , wherein the image processor is further configured to determine the local color-related ratio for said color pixel by deriving said local color-related ratio from the pixel values of one or more color pixels having the same color as said color pixel, and that are positioned adjacent to said at least one IR pixel in the array of pixels.

16. The imaging system of claim 12 , wherein the image processor is further configured to determine said local color-related ratio for said color pixel by:

determining a plurality of color-related ratios respectively related to a plurality of IR pixels positioned adjacent to said color pixel in the array of pixels; and

averaging said plurality of color-related ratios.

17. A non-transitory computer readable media comprising instructions that, when executed by a computer, cause the computer to:

process pixel values provided at an output of an image sensor comprising a two-dimensional array of pixels including a mix of IR pixels and color pixels, wherein each of the color pixels are processed by:

determining a local color-related ratio indicative of a part of infrared light reception in the color pixel; and

estimating an infrared portion value from a color pixel value based on the local color-related ratio;

generate an infrared image from pixel values from the IR pixels and the estimated infrared portion values from the color pixels; and

prior to processing, perform a first preprocess including adjusting an infrared transmittance of the color pixels in the array of pixels to a target level by applying color-related normalization factors to the color pixel values, the color-related normalization factors being determined by:

illuminating with an only infrared light signal a target that is monotonically reflective in a light spectrum from visible spectrum to infrared spectrum;

receiving by the image sensor the infrared light signal reflected by said target;

averaging all IR pixel values to determine a mean IR value;

for each color channel, averaging all color pixel values to determine a mean color value; and

calculating each color-related normalization factor by calculating the ratio between the mean IR value and the mean color value.

Assignments (4)
MERGER Recorded Feb 11, 2024
From: APTIV TECHNOLOGIES (2) S.À R.L.
To: APTIV MANUFACTURING MANAGEMENT SERVICES S.À R.L.
Reel/Frame 066566/0173 →
ENTITY CONVERSION Recorded Feb 11, 2024
From: APTIV TECHNOLOGIES LIMITED
To: APTIV TECHNOLOGIES (2) S.À R.L.
Reel/Frame 066746/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 11, 2024
From: APTIV MANUFACTURING MANAGEMENT SERVICES S.À R.L.
To: APTIV TECHNOLOGIES AG
Reel/Frame 066551/0219 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 29, 2022
From: SMITS, SOEREN; PITZ, ULRICH; PAZEKHA, STEPAN
To: APTIV TECHNOLOGIES LIMITED
Reel/Frame 060924/0332 →
Priority Claims (1)
EP 22186391 · Jul 22, 2022 · regional
Continuity (2)
Provisional Application 63236430 · Aug 24, 2021
Related Publication 20230074718A1 · Mar 9, 2023