Composite image signal processor
Systems and techniques are described for image processing. An imaging system can include an image sensor that captures image data. An image signal processor (ISP) of the imaging system can demosaic the image data. The imaging system can input the image data into one or more trained machine learning models, in some cases along with metadata associated with the image data. The one or more trained machine learning models can output settings for a set of parameters of the ISP based on the image data and/or the metadata. The imaging system can generate an output image by processing the image data using the ISP, with the parameters of the ISP set according to the settings. Each pixel of the pixels of the image data can be processed using a respective setting for adjusting a corresponding parameter. The parameters of the ISP can include gain, offset, gamma, and Gaussian filtering.
1. An apparatus for processing image data, the apparatus comprising:
at least one memory; and
at least one processor coupled to the at least one memory, the at least one processor configured to:
obtain image data having at least a first color channel and a second color channel;
determine a plurality of settings for adjusting at least one parameter of an image signal processor (ISP) based on an output of at least one trained machine learning model that uses the image data as an input, wherein the plurality of settings include a first setting associated with the first color channel and a second setting associated with the second color channel; and
process the image data using the ISP to generate an output image frame, wherein the first setting is used to adjust the at least one parameter for the first color channel to process the image data, wherein the second setting is used to adjust the at least one parameter for the second color channel to process the image data.
2. The apparatus of claim 1 , wherein the first setting is used to adjust the at least one parameter for the first color channel without using the second setting for the first color channel, wherein the second setting is used to adjust the at least one parameter for the second color channel without using the first setting for the second color channel.
3. The apparatus of claim 1 , wherein the image data also has a third color channel, wherein the plurality of settings include a third setting associated with the third color channel, wherein the third setting is used to adjust the at least one parameter for the third color channel to process the image data.
4. The apparatus of claim 3 , wherein first color channel is a red color channel, wherein second color channel is a green color channel, and wherein third color channel is a blue color channel.
5. The apparatus of claim 1 , wherein first color channel represents a frequency range within a visible light spectrum, wherein the second color channel represents a frequency range outside of the visible light spectrum.
6. The apparatus of claim 1 , wherein the plurality of settings spatially vary across different pixels of an image frame associated with the image data.
7. The apparatus of claim 1 , the at least one processor configured to:
demosaic the image data to process the image data using the ISP to generate the output image frame.
8. The apparatus of claim 7 , wherein the first color channel corresponds to a first color of a color filter array of an image sensor that captures the image data, wherein the second color channel corresponds to a second color of the color filter array of the image sensor that captures the image data.
9. The apparatus of claim 1 , wherein the image data includes demosaiced image data.
10. The apparatus of claim 1 , the at least one processor configured to:
receive the image data from an image sensor to obtain the image data.
11. The apparatus of claim 1 , wherein the at least one trained machine learning model also uses metadata associated with the image data as a second input to generate the output.
12. The apparatus of claim 11 , wherein the metadata includes at least one image capture setting used to capturing the image data.
13. The apparatus of claim 1 , wherein the first setting associated with the first color channel includes a first gain associated with the first color channel, wherein the second setting associated with the second color channel includes a second gain associated with the second color channel.
14. The apparatus of claim 1 , wherein the first setting associated with the first color channel includes a first offset associated with the first color channel, wherein the second setting associated with the second color channel includes a second offset associated with the second color channel.
15. The apparatus of claim 1 , the at least one processor configured to:
determine an additional setting for adjusting a second parameter of the image signal processor (ISP), wherein the additional setting is used to adjust the second parameter for both the first color channel and the second color channel to process the image data.
16. The apparatus of claim 15 , wherein the second parameter is associated with tone adjustment.
17. The apparatus of claim 15 , wherein the second parameter is associated with blurring.
18. The apparatus of claim 1 , wherein the at least one parameter is associated with tone adjustment.
19. The apparatus of claim 1 , wherein the at least one parameter is associated with color saturation.
20. The apparatus of claim 1 , wherein the at least one parameter is associated with blurring.
21. The apparatus of claim 1 , the at least one processor configured to:
further train the at least one trained machine learning model based on at least the output image frame.
22. A method for processing image data, the method comprising:
obtaining image data having at least a first color channel and a second color channel;
determining a plurality of settings for adjusting at least one parameter of an image signal processor (ISP) based on an output of at least one trained machine learning model that uses the image data as an input, wherein the plurality of settings include a first setting associated with the first color channel and a second setting associated with the second color channel; and
processing the image data using the ISP to generate an output image frame, wherein the first setting is used to adjust the at least one parameter for the first color channel to process the image data, wherein the second setting is used to adjust the at least one parameter for the second color channel to process the image data.
23. The method of claim 22 , wherein the first setting is used to adjust the at least one parameter for the first color channel without using the second setting for the first color channel, wherein the second setting is used to adjust the at least one parameter for the second color channel without using the first setting for the second color channel.
24. The method of claim 22 , wherein the image data also has a third color channel, wherein the plurality of settings include a third setting associated with the third color channel, wherein the third setting is used to adjust the at least one parameter for the third color channel to process the image data.
25. The method of claim 22 , wherein the plurality of settings spatially vary across different pixels of an image frame associated with the image data.
26. The method of claim 22 , further comprising:
demosaicing the image data to process the image data using the ISP to generate the output image frame.
27. The method of claim 22 , further comprising:
receiving the image data from an image sensor to obtain the image data.
28. The method of claim 22 , wherein the at least one trained machine learning model also uses metadata associated with the image data as a second input to generate the output.
29. The method of claim 22 , wherein the first setting associated with the first color channel includes a first gain associated with the first color channel, wherein the second setting associated with the second color channel includes a second gain associated with the second color channel.
30. The method of claim 22 , wherein the first setting associated with the first color channel includes a first offset associated with the first color channel, wherein the second setting associated with the second color channel includes a second offset associated with the second color channel.