Multispectral image decorrelation method and system
A method of creating a multispectral decorrelation model for use in determining a visible image from a multispectral image captured using a multispectral image sensor, the method comprising the steps of: generating, using a plurality of quantum efficiency curves for the multispectral image sensor and a plurality of synthetic light spectrum vectors, a grid of synthetic multispectral pixel values and a corresponding grid of synthetic visible pixel values, wherein each synthetic visible pixel value is substantially decorrelated from a non-visible component of a corresponding synthetic multispectral pixel value; and determining a multispectral decorrelation model using the grid of synthetic multispectral pixel values and the corresponding grid of synthetic visible pixel values, wherein the multispectral decorrelation model in use maps a multispectral pixel value of the multispectral image to a visible pixel value of the visible image.
1. A method of creating a multispectral decorrelation model for use in determining a visible image from a multispectral image captured using a multispectral image sensor, the method comprising the steps of:
generating, using a plurality of quantum efficiency curves for the multispectral image sensor and a plurality of synthetic light spectrum vectors, a grid of synthetic multispectral pixel values and a corresponding grid of synthetic visible pixel values, wherein each synthetic visible pixel value is substantially decorrelated from a non-visible component of a corresponding synthetic multispectral pixel value; and
determining a multispectral decorrelation model using the grid of synthetic multispectral pixel values and the corresponding grid of synthetic visible pixel values, wherein the multispectral decorrelation model, in use, maps a multispectral pixel value of the multispectral image to a visible pixel value of the visible image.
2. The method of claim 1 , wherein the plurality of synthetic light spectrum vectors evenly sample multispectral image space when point-wise multiplied by the plurality of quantum efficiency curves.
3. The method of claim 1 , wherein the multispectral decorrelation model consists of a trainable model such that a trained multispectral decorrelation model estimates a transformation from multispectral image space to visible image space.
4. The method of claim 3 , wherein the multispectral decorrelation model is one of: a Gaussian process regression model; a lattice regression model; a neural network; a genetic algorithm.
5. The method of claim 3 , wherein the multispectral decorrelation model is exposure invariant.
6. The method of claim 3 , further comprising the steps of:
determining a plurality of input multispectral pixel values by evenly sampling multispectral image space; and
providing the plurality of input multispectral pixel values to the trained multispectral decorrelation model thereby to populate a decorrelation look-up-table.
7. The method of claim 6 , further comprising the step of:
saving the decorrelation look-up-table to a computer-readable medium.
8. The method of claim 1 , wherein the grid of synthetic multispectral pixel values is generated by point-wise multiplying the plurality of quantum efficiency curves by the plurality of synthetic light spectrum vectors.
9. The method of claim 1 , wherein the grid of corresponding synthetic visible pixel values is generated by one of: point-wise multiplying the plurality of quantum efficiency curves by the plurality of synthetic light spectrum vectors in the range of 400-700 nm; and point-wise multiplying a plurality of visible light specific quantum efficiency curves by the plurality of synthetic light spectrum vectors.
10. The method of claim 1 , wherein a synthetic light spectrum vector of the plurality of synthetic light spectrum vectors represents a spectrum of light incident on the multispectral image sensor.
11. The method of claim 1 , further comprising the step of:
optimizing, using a numerical optimization algorithm, a loss function thereby to generate a synthetic light spectrum vector of the plurality of synthetic light spectrum vectors.
12. The method of claim 11 , wherein the loss function calculates a difference between a candidate synthetic light spectrum vector multiplied by the plurality of quantum efficiency curves and a target multispectral pixel value.
13. The method of claim 11 , wherein the numerical optimization algorithm is one of: a conjugate gradient method; and a quadratic programming algorithm.
14. The method of claim 1 , further comprising the step of:
multiplying a transmission spectrum of an optical filter to each quantum efficiency curve of the plurality of quantum efficiency curves prior to the step of generating.
15. The method of claim 14 , wherein the optical filter is a dual band-pass filter comprising an infrared transmission centered at 850 nm or 940 nm.
16. The method of claim 1 , wherein a multispectral pixel value of the multispectral image comprises a red pixel value, a blue pixel value, a green pixel value, and an infrared pixel value.
17. The method of claim 1 , wherein a multispectral pixel value of the multispectral image comprises a red pixel value, a blue pixel value, a green pixel value, and a white pixel value.
18. An apparatus comprising:
one or more processors; and
a computer-readable medium including one or more sequences of instructions that, when executed by the one or more processors, cause the apparatus to execute the method of claim 1 .
19. The apparatus of claim 18 , wherein the sequences of instructions further cause the apparatus to execute the method of claim 2 .
20. A non-transitory computer-readable medium having stored thereon sequences of instructions to cause one or more processors to carry out the method of claim 1 .