Method and system for joint demosaicking and spectral signature estimation
Methods and systems for determining parameters of a desired target image from hyperspectral imagery obtained from a surgical procedure are described herein. Techniques may include capturing in real time hyperspectral snapshot mosaic images of a scene using a hyperspectral image sensor coupled to an optical scope, the snapshot mosaic images being of relatively low spatial and low spectral resolution; undertaking spatio-spectrally aware demosaicking of the snapshot mosaic images, the demosaicking comprising upsampling of the snapshot mosaic images and the application of a spectral calibration operator, to generate a virtual hypercube of the snapshot mosaic image data, the virtual hypercube comprising image data of relatively high spatial resolution compared to the snapshot mosaic images; from the image data in the virtual hypercube, determining relatively high spatial resolution parameters of a desired target image; and outputting in real time the determined relatively high-resolution parameters as representative of the desired target image.
1. A method of determining parameters of a desired target image from hyperspectral imagery obtained from a surgical procedure, comprising:
capturing in real time hyperspectral snapshot mosaic images of a scene from the surgical procedure using a hyperspectral image sensor coupled to an optical scope, the snapshot mosaic images being of relatively low spatial and low spectral resolution;
undertaking spatio-spectrally aware demosaicking of the snapshot mosaic images, the demosaicking comprising an algorithm to optimise for a cost function accounting for spatial crosstalk across neighbouring pixels corresponding to different spectral bands in the snapshot mosaic image, upsampling of the snapshot mosaic images, and the application of a spectral calibration operator, to generate a virtual hypercube of the snapshot mosaic image data, the virtual hypercube comprising image data of relatively high spatial resolution compared to the snapshot mosaic images;
from the image data in the virtual hypercube, determining relatively high spatial resolution parameters of a desired target image; and
outputting in real time the determined relatively high-resolution parameters as representative of the desired target image.
2. A method according to claim 1 , wherein the demosaicking comprises machine learning.
3. A method according to claim 1 , wherein the demosaicking is temporally consistent between two or more consecutive frames based on motion compensation in between frames.
4. A method according to claim 1 , and further comprising, prior to capturing the hyperspectral snapshot mosaic images, undertaking a white balancing operation on the hyperspectral image sensor.
5. A method according to claim 4 , wherein the white balancing operation comprises separately acquiring reference images, including dark and white reference mosaic images w d;τ d and w w;τ w at integration times τ d and τ w , respectively, and deploying a linear model where in addition to the acquired mosaic image w τ of an object with integration time τ, a white reference mosaic image w w;τ w of a reflectance tile with integration time τ w , and dark reference mosaic images w d;τ and w d;τ w are acquired with integration times τ and τ w , with a closed shutter, and the white balancing operation yields a reflectance mosaic image given by
r
:=
w
τ
-
w
d
;
τ
w
w
;
τ
w
-
w
d
;
τ
w
τ
w
τ
∈
W
.
6. A method according to claim 1 , and further comprising, prior to capturing the hyperspectral snapshot mosaic images, undertaking a spatiospectral calibration operation on the hyperspectral image sensor.
7. A method according to claim 6 , wherein a real spectral filter response operator B F : → and a spatial cross-talk operator T: W→W are estimated in a controlled setup to account for parasitical effects during image acquisition.
8. A method according to claim 7 , and further comprising measuring a characteristic of the hyperspectral image sensor to obtain a measured system filter response operator A F meas : U→W by acquiring snapshot mosaic image data using collimated light and sweeping through all wavelengths in conjunction with an imaging target with known, typically spatially-constant, spectral signature.
9. A method according to claim 1 , wherein the determining of the relatively high spatial parameters further comprises analysing pixel-level hyperspectral information for its composition of unique end-members characterised by specific spectral signatures.
10. A method according to claim 1 , wherein the determining of the relatively high spatial parameters further comprises estimation of tissue properties per spatial location (typically pixels) from reflectance information of hyperspectral imaging, such as pixel-level tissue absorption information.
11. A method of determining parameters of a desired target image from hyperspectral imagery, comprising:
capturing in real time hyperspectral snapshot mosaic images of a scene from a surgical procedure using a hyperspectral image sensor coupled to an optical scope, the snapshot mosaic images being of relatively low spatial and low spectral resolution;
undertaking a joint demosaicking and parameter estimation from the snapshot mosaic images to determine relatively high spatial resolution parameters of a desired target image; and
outputting in real time the determined relatively high-resolution parameters as representative of the desired target image,
wherein the demosaicking is spatio-spectrally aware, and comprises an algorithm to optimise for a cost function accounting for spatial crosstalk across neighbouring pixels corresponding to different spectral bands in the snapshot mosaic image, upsampling of the snapshot mosaic images, and application of a spectral calibration operator.
12. A non-transitory computer readable storage medium storing a computer program that when executed causes a hyperspectral imaging system to perform a method comprising:
capturing in real time hyperspectral snapshot mosaic images of a scene from a surgical procedure using a hyperspectral image sensor coupled to an optical scope, the snapshot mosaic images being of relatively low spatial and low spectral resolution;
undertaking spatio-spectrally aware demosaicking of the snapshot mosaic images, the demosaicking comprising an algorithm to optimise for a cost function accounting for spatial crosstalk across neighbouring pixels corresponding to different spectral bands in the snapshot mosaic image, upsampling of the snapshot mosaic images, and the application of a spectral calibration operator, to generate a virtual hypercube of the snapshot mosaic image data, the virtual hypercube comprising image data of relatively high spatial resolution compared to the snapshot mosaic images;
from the image data in the virtual hypercube, determining relatively high spatial resolution parameters of a desired target image; and
outputting in real time the determined relatively high-resolution parameters as representative of the desired target image.