RGB-IR pixel pattern conversion via conversion engine
Disclosed herein are improvements to pixel pattern conversion, upsampling, and IR decontamination processes. An example includes an image processing pipeline comprising an upstream component, a pattern conversion component downstream with respect to the upstream component in the image processing pipeline, and a downstream component relative to the pattern conversion component. The pattern conversion component is configured to obtain RGB-IR pixel data produced by the upstream component of the image processing pipeline and convert the RGB-IR pixel data into RGB pixel data and IR pixel data using a conversion engine. The conversion engine is configured to demosaic the RGB-IR pixel into the RGB pixel data and the IR pixel data, remosaic the RGB pixel data into an RGB pattern and the IR pixel data into an IR pattern and remove IR contamination from the RGB pixel data of the RGB pattern for use by the downstream component.
1. An image processing pipeline, comprising:
an upstream component;
a pattern conversion component downstream with respect to the upstream component in the image processing pipeline; and
a downstream component relative to the pattern conversion component;
wherein the pattern conversion component is configured to:
obtain red, green, blue and infrared (RGB-IR) pixel data produced by the upstream component of the image processing pipeline;
convert the RGB-IR pixel data into red, green and blue (RGB) pixel data and infrared (IR) pixel data using a conversion engine configured to:
demosaic the RGB-IR pixel data into the RGB pixel data and the IR pixel data;
remosaic the RGB pixel data into a RGB pattern and the IR pixel data into an IR pattern; and
remove IR contamination from the RGB pixel data of the RGB pattern; and
supply the RGB pattern and the IR pattern to the downstream component of the image processing pipeline.
2. The image processing pipeline of claim 1 , wherein the conversion engine is configured to demosaic the RGB-IR pixel data into the RGB pixel data and the IR pixel data by:
for each pixel of the RGB-IR pixel data:
identifying a context of the pixel;
interpolating, based on the context of the pixel, one of a red, green, or blue pixel value for the RGB pixel data; and
interpolating, based on the context of the pixel, an IR pixel value for the IR pixel data.
3. The image processing pipeline of claim 2 , wherein the context comprises information of the RGB-IR pixel data at a location associated with a pixel and information of neighboring pixels, with respect to the pixel, of the RGB-IR pixel data.
4. The image processing pipeline of claim 2 , wherein the conversion engine is configured to remosaic the RGB pixel data into the RGB pattern and the IR pixel data into the IR pattern by:
mapping the interpolated red, green, and blue pixel values to locations in the RGB pattern; and
mapping the interpolated IR pixel values to locations in the IR pattern.
5. The image processing pipeline of claim 2 , wherein the conversion engine is further configured to determine, for each pixel of the RGB-IR pixel data, a first amount of the IR contamination of the pixel.
6. The image processing pipeline of claim 5 , wherein the conversion engine is further configured to determine, for each pixel of the RGB-IR pixel data, whether an interpolated pixel value of the pixel exceeds a threshold value.
7. The image processing pipeline of claim 6 , wherein the conversion engine is further configured to, for each pixel of the RGB-IR pixel data, subtract the first amount of the IR contamination from the interpolated pixel value of the pixel based on the interpolated pixel value not exceeding the threshold value, and subtract a second amount of the IR contamination from the interpolated pixel value based on the interpolated pixel value exceeding the threshold value, wherein the second amount is less than the first amount.
8. A pattern conversion component in an image processing pipeline, comprising:
an interface; and
circuitry coupled to the interface;
wherein the interface is configured to communicate with an upstream component and a downstream component of the image processing pipeline relative to the pattern conversion component; and
wherein the circuitry is configured to:
obtain, via the interface, red, green, blue and infrared (RGB-IR) pixel data produced by the upstream component;
convert the RGB-IR pixel data into red, green and blue (RGB) pixel data and infrared (IR) pixel data using a conversion engine configured to:
demosaic the RGB-IR pixel data into the RGB pixel data and the IR pixel data;
remosaic the RGB pixel data into a RGB pattern and the IR pixel data into an IR pattern; and
remove IR contamination from the RGB pixel data of the RGB pattern; and
supply, via the interface, the RGB pattern and the IR pattern to the downstream component.
9. The pattern conversion component of claim 8 , wherein the circuitry is configured to demosaic the RGB-IR pixel data into the RGB pixel data and the IR pixel data by:
for each pixel of the RGB-IR pixel data:
identifying a context of the pixel;
interpolating, based on the context of the pixel, one of a red, green, or blue pixel value for the RGB pixel data; and
interpolating, based on the context of the pixel, an IR pixel value for the IR pixel data.
10. The pattern conversion component of claim 9 , wherein the context comprises information of the RGB-IR pixel data at a location associated with a pixel and information of neighboring pixels, with respect to the pixel, of the RGB-IR pixel data.
11. The pattern conversion component of claim 9 , wherein the circuitry is configured to remosaic the RGB pixel data into the RGB pattern and the IR pixel data into the IR pattern by:
mapping the interpolated red, green, and blue pixel values to locations in the RGB pattern; and
mapping the interpolated IR pixel values to locations in the IR pattern.
12. The pattern conversion component of claim 9 , wherein the circuitry is further configured to determine, for each pixel of the RGB-IR pixel data, a first amount of the IR contamination of the pixel.
13. The pattern conversion component of claim 9 , wherein the circuitry is further configured to determine, for each pixel of the RGB-IR pixel data, whether an interpolated pixel value of the pixel exceeds a threshold value.
14. The pattern conversion component of claim 13 , wherein the circuitry is further configured to, for each pixel of the RGB-IR pixel data, subtract a first amount of the IR contamination from the interpolated pixel value of the pixel based on the interpolated pixel value not exceeding the threshold value, and subtract a second amount of the IR contamination from the interpolated pixel value based on the interpolated pixel value exceeding the threshold value, wherein the second amount is less than the first amount.
15. A method of operating a pattern conversion component of an image processing pipeline, the method comprising:
obtaining red, green, blue and infrared (RGB-IR) pixel data produced by a first component;
converting the RGB-IR pixel data into red, green and blue (RGB) pixel data and infrared (IR) pixel data by:
demosaicing the RGB-IR pixel data into the RGB pixel data and the IR pixel data;
remosaicing the RGB pixel data into a RGB pattern and the IR pixel data into an IR pattern; and
removing IR contamination from the RGB pixel data of the RGB pattern; and
supplying the RGB pattern and IR pattern to a second component.
16. The method of claim 15 , wherein demosaicing the RGB-IR pixel data into the RGB pixel data and the IR pixel data comprises:
for each pixel of the RGB-IR pixel data:
identifying a context of the pixel;
interpolating, based on the context of the pixel, one of a red, green, or blue pixel value for the RGB pixel data; and
interpolating, based on the context of the pixel, an IR pixel value for the IR pixel data.
17. The method of claim 16 , wherein the context comprises information of the RGB-IR pixel data at a location associated with a pixel and information of neighboring pixels, with respect to the pixel, of the RGB-IR pixel data.
18. The method of claim 16 , wherein remosaicing the RGB pixel data into the RGB pattern and the IR pixel data into the IR pattern comprises:
mapping the interpolated red, green, and blue pixel values to locations in the RGB pattern; and
mapping the interpolated IR pixel values to locations in the IR pattern.
19. The method of claim 16 , further comprising, determining, for each pixel of the RGB-IR pixel data, a first amount of the IR contamination of a given pixel.
20. The method of claim 19 , further comprising, subtracting, for each pixel of the RGB-IR pixel data, the first amount of the IR contamination from an interpolated pixel value of the pixel based on the interpolated pixel value not exceeding a threshold value, and subtracting a second amount of the IR contamination from the interpolated pixel value based on the interpolated pixel value exceeding the threshold value, wherein the second amount is less than the first amount.