Image reconstruction from image sensor output

1. An apparatus, comprising:

a plurality of sensing units, each sensing unit comprising one or more photodiodes, each of the one or more photodiodes being associated with a channel having a target wavelength range for photon-to-charge conversion; and

an image processor configured to:

receive input data generated by the photodiodes of the plurality of sensing units responsive to incident light;

obtain, for each channel, a plurality of channel coefficients, the plurality of channel coefficients being configured to, when combined with the input data to generate channel output data for the each channel, increase a main component of the channel output data contributed by a part of the incident light within the target wavelength range of the each channel with respect to a crosstalk component of the channel output data contributed by a part of the incident light out of the target wavelength range, wherein the plurality of channel coefficients, when combined with a raw spectral response curve of each of the one or more photodiodes to obtain a target spectral response curve of the photodiode, maximizes a first portion of the target spectral response curve within the target wavelength range of each channel with respect to a second portion of the target spectral response curve out of the target wavelength range of each channel; and

generate, for the each channel, the channel output data based on combining the input data with the plurality of channel coefficients to reconstruct an image for the each channel.

2. The apparatus of claim 1 , further comprising a calibration module configured to:

obtain data representing the raw spectral response curve of each photodiode of the photodiodes, the raw spectral response curve representing a spectral response of the each photodiode operating in the respective sensing unit to light of a wavelength range, the data representing the raw spectral response curves being combinable with the plurality of channel coefficients of the each channel to generate the target spectral response curve for the each channel; and

determine, for the each channel, a plurality of channel separation coefficients that maximizes the first portion of the target spectral response curve within the target wavelength range of the each channel with respect to the second portion of the target spectral response curve out of the target wavelength range of the each channel,

wherein the plurality of channel separation coefficients for the each channel is part of the plurality of channel coefficients for the each channel.

3. The apparatus of claim 2 , wherein the raw spectral response curve of each photodiode includes a first portion within the target wavelength of the channel associated with the each photodiode and a second portion out of the target wavelength out of the channel associated with the each photodiode;

wherein the calibration module is configured to, for the each channel:

determine, for each raw spectral response of the raw spectral responses, a first portion that is within the target wavelength range of the each channel and a second portion that is outside the target wavelength range;

determine an in-band matrix based on correlations among the first portions of the raw spectral responses;

determine an out-of-band matrix based on correlations among the second portions of the raw spectral responses; and

determine the plurality of channel separation coefficients that increases a first combination of the in-band matrix and the plurality of channel separation coefficients with respect to a second combination of the out-of-band matrix and the plurality of channel separation coefficients.

4. The apparatus of claim 3 , wherein the first combination comprises a matrix multiplication of a transpose of the plurality of channel separation coefficients, the in-band matrix, and the plurality of channel separation coefficients; and

wherein the second combination comprises a matrix multiplication of the transpose of the plurality of channel separation coefficients, the out-of-band matrix, and the plurality of channel separation coefficients.

5. The apparatus of claim 4 , wherein the determination of the plurality of channel coefficients comprises solving a generalized-Eigenvalue problem of maximizing a ratio between the first combination and the second combination.

6. The apparatus of claim 5 , wherein the calibration module is configured to:

determine, for the each channel, the target spectral response curve based on combining the raw spectral response curves with the plurality of channel coefficients of the each channel; and

determine, for the each channel, a plurality of channel crosstalk coefficients that indicate crosstalk components of the channel output data of the each channel contributed by the photodiodes;

wherein the plurality of channel crosstalk coefficients of the each channel is determined based on minimizing a difference between the spectral response curve of the each channel and a combination of the plurality of channel crosstalk coefficients of the each channel and the target spectral response curves; and

wherein the plurality of channel separation coefficients for the each channel is part of the plurality of channel coefficients for the each channel.

7. The apparatus of claim 2 , wherein the image processor is configured to:

generate pre-processed input data for the each channel based on performing a demosaicing operation on the input data; and

generate, for the each channel, the channel output data based on combining the pre-processed input data of the each channel with the plurality of channel separation coefficients of the each channel to reconstruct an image for the each channel.

8. The apparatus of claim 7 , wherein the image processor is configured to:

determine, based on the input data, a first image model associated with a visible light wavelength range and a second image model associated with an infra-red light wavelength range; and

generate the channel output data for the each channel based on minimizing, under a pre-determined constraint, a first difference between the first image model and a first combination of a first part of the channel output data associated with the visible light wavelength range and the plurality of channel crosstalk coefficients of the each channel and based on minimizing, under the pre-determined constraint, a second difference between the second image model and a second combination of a second part of the channel output data associated with the infra-red light wavelength range and the plurality of channel crosstalk coefficients of the each channel.

9. The apparatus of claim 8 , wherein the pre-determined constraint comprises a constraint on an image gradient.

10. The apparatus of claim 8 , wherein the calibration module is configured to generate the plurality of channel crosstalk coefficients in Fourier domain; and

wherein the image processor is configured to:

generate the pre-processed input data in the Fourier domain;

generate an image formation model in the Fourier domain; and

generate the channel output data for the each channel based on minimizing a difference between the image formation model in the Fourier domain and a combination of the channel output data and the plurality of channel crosstalk coefficients of the each channel in the Fourier domain under the pre-determined constraint.

11. The apparatus of claim 1 , wherein each sensing unit of the plurality of sensing units comprises a first photodiode and a second photodiode forming a stack along a direction perpendicular to a light receiving surface of the each sensing unit.

12. The apparatus of claim 1 , wherein each sensing unit of the plurality of sensing units comprises a single photodiode.

13. A method, comprising:

receiving input data generated by photodiodes of a plurality of sensing units responsive to incident light, wherein each sensing unit comprises one or more photodiodes, each of the one or more photodiodes being associated with a channel having a target wavelength range for photon-to-charge conversion;

obtaining, for each channel, a plurality of channel coefficients, the plurality of channel coefficients being configured to, when combined with the input data to generate channel output data for the each channel, increase a main component of the channel output data contributed by a part of the incident light within the target wavelength range of the each channel with respect to a crosstalk component of the channel output data contributed by a part of the incident light out of the target wavelength range, wherein the plurality of channel coefficients, when combined with a raw spectral response curve of each of the one or more photodiodes to obtain a target spectral response curve of the photodiode, maximizes a first portion of the target spectral response curve within the target wavelength range of each channel with respect to a second portion of the target spectral response curve out of the target wavelength range of each channel; and

generating, for the each channel, the channel output data based on combining the input data with the plurality of channel coefficients to reconstruct an image for the each channel.

14. The method of claim 13 , further comprising:

obtaining data representing the raw spectral response curve of each photodiode of the photodiodes, the raw spectral response curve representing a spectral response of the each photodiode operating in the respective sensing unit to light of a wavelength range, the data representing the raw spectral response curves being combinable with the plurality of channel coefficients of the each channel to generate a target spectral response curve for the each channel; and

determining, for the each channel, a plurality of channel separation coefficients that maximizes a first portion of the target spectral response curve within the target wavelength range of the each channel with respect to a second portion of the target spectral response curve out of the target wavelength range of the each channel,

wherein the plurality of channel separation coefficients for the each channel is part of the plurality of channel coefficients for the each channel.

15. The method of claim 14 , wherein the raw spectral response curve of each photodiode includes a first portion within the target wavelength of the channel associated with the each photodiode and a second portion out of the target wavelength out of the channel associated with the each photodiode;

wherein the method further comprises, for the each channel:

determining, for each raw spectral response of the raw spectral responses, a first portion that is within the target wavelength range of the each channel and a second portion that is outside the target wavelength range;

determining an in-band matrix based on correlations among the first portions of the raw spectral responses;

determining an out-of-band matrix based on correlations among the second portions of the raw spectral responses; and

determining the plurality of channel separation coefficients that increases a first combination of the in-band matrix and the plurality of channel separation coefficients with respect to a second combination of the out-of-band matrix and the plurality of channel separation coefficients.

16. The method of claim 15 , wherein the first combination comprises a matrix multiplication of a transpose of the plurality of channel separation coefficients, the in-band matrix, and the plurality of channel separation coefficients; and

wherein the second combination comprises a matrix multiplication of the transpose of the plurality of channel separation coefficients, the out-of-band matrix, and the plurality of channel separation coefficients.

17. The method of claim 16 , wherein determining the plurality of channel coefficients comprises solving a generalized-Eigenvalue problem of maximizing a ratio between the first combination and the second combination.

18. The method of claim 17 , further comprising:

determining, for the each channel, the target spectral response curve based on combining the raw spectral response curves with the plurality of channel coefficients of the each channel; and

determining, for the each channel, a plurality of channel crosstalk coefficients that indicate crosstalk components of the channel output data of the each channel contributed by the plurality of photodiodes;

wherein the plurality of channel crosstalk coefficients of the each channel is determined based on minimizing a difference between the spectral response curve of the each channel and a combination of the plurality of channel crosstalk coefficients of the each channel and the target spectral response curves; and

wherein the plurality of channel separation coefficients for the each channel is part of the plurality of channel coefficients for the each channel.

19. The method of claim 18 , further comprising:

determining, based on the input data, a first image model associated with a visible light wavelength range and a second image model associated with an infra-red light wavelength range; and

generating the channel output data for the each channel based on minimizing, under a pre-determined constraint, a first difference between the first image model and a first combination of a first part of the channel output data associated with the visible light wavelength range and the plurality of channel crosstalk coefficients of the each channel and based on minimizing, under the pre-determined constraint, a second difference between the second image model and a second combination of a second part of the channel output data associated with the infra-red light wavelength range and the plurality of channel crosstalk coefficients of the each channel.

20. A non-transitory computer readable medium storing instructions that, when executed by a hardware processor, causes the hardware processor to perform:

receiving input data generated by photodiodes of a plurality of sensing units responsive to incident light, wherein each sensing unit comprises one or more photodiodes, each of the one or more photodiodes being associated with a channel having a target wavelength range for photon-to-charge conversion;

obtaining, for each channel, a plurality of channel coefficients, the plurality of channel coefficients being configured to, when combined with the input data to generate channel output data for the each channel, increase a main component of the channel output data contributed by a part of the incident light within the target wavelength range of the each channel with respect to a crosstalk component of the channel output data contributed by a part of the incident light out of the target wavelength range, wherein the plurality of channel coefficients, when combined with a raw spectral response curve of each of the one or more photodiodes to obtain a target spectral response curve of the photodiode, maximizes a first portion of the target spectral response curve within the target wavelength range of each channel with respect to a second portion of the target spectral response curve out of the target wavelength range of each channel; and

generating, for the each channel, the channel output data based on combining the input data with the plurality of channel coefficients to reconstruct an image for the each channel.