Optical method
A method of reconstructing a hyperspectral data cube from an encoded and/or spectrally sheared image and an unencoded and spectrally unsheared image of a scene comprises providing the encoded and/or spectrally sheared image and an unencoded and spectrally unsheared image to a neural network comprising an encoder-decoder structure having an encoder portion and a decoder portion; performing one or more primary convolution operations using the encoder portion; and performing one or more transpose convolution operations using the decoder portion; performing spatially adaptive normalization and/or inverted attention normalization on an output of primary convolution operations and/or transpose convolution operations wherein the spatially adaptive normalization and/or inverted attention normalization is performed using a representation of an encoding scheme used to encode the at least one encoded and/or spectrally sheared image and an unencoded and spectrally unsheared image of the scene.
1 . A method of reconstructing a hyperspectral data cube from at least one encoded and/or spectrally sheared image and an unencoded and spectrally unsheared image of a scene, the method comprising: providing the at least one encoded and/or spectrally sheared image and an unencoded and spectrally unsheared image to a neural network comprising an encoder-decoder structure having an encoder portion and a decoder portion; performing one or more primary convolution operations using the encoder portion; and performing one or more transpose convolution operations using the decoder portion; performing spatially adaptive normalization and/or inverted attention normalization on an output of at least one of the one or more primary convolution operations and/or transpose convolution operations, wherein the spatially adaptive normalization and/or inverted attention normalization is performed using a representation of an encoding scheme used to encode the at least one encoded and/or spectrally sheared image and an unencoded and spectrally unsheared image of the scene.
2 . The method of claim 1 , wherein performing spatially adaptive normalization comprises:
performing a first secondary convolution operation on a representation of a mask or unencoded and spectrally unsheared image of the scene;
performing a second secondary convolution operation on an output of the first secondary convolution operation;
performing element-wise multiplication between an output of the second secondary convolution operation and an output of a primary convolution operation or transpose convolution operation; and
summing an output of the second secondary convolution operation with an output of the element-wise multiplication.
3 . The method of claim 1 , wherein performing inverted attention normalization comprises a convolution or transpose convolution operation applied on to input features, and/or one or more convolutional operations successively applied to results of the spatially adaptive normalization.
4 . The method of claim 2 , wherein performing inverted attention normalization comprises performing one of a sigmoid operation and/or softmax activation operation applied to the output of the primary convolutions or the secondary convolution operations previously applied, wherein the method comprises performing an element wise multiplication between an output of the primary convolutional operation and a result of the softmax activation operation, and one or more convolution and/or transposed convolution operations successively applied to the result of the element wise multiplication.
5 . The method of claim 1 , further comprising resizing a representation of a mask to match a resolution of the output of primary convolution operations or transpose convolution operations.
6 . The method of claim 1 , further comprising performing one or more residual convolution operations using one or both of the encoder portion and the decoder portion.
7 . The method of claim 6 , where performing a residual convolution operation comprises:
providing an input;
performing one or more successive convolution operations; and
combining the input and an output of the series of successive convolution operations.
8 . The method of claim 7 , wherein combining the input and the output of the series of successive convolution operations comprises combining the input and the output of the series of successive convolution operations by either summation, concatenation or multiplication or a combination of these operations.
9 . The method of claim 7 , wherein the input is one of:
an output of a primary convolution operation or a transpose convolution operation;
an output of spatially adaptive normalization.
10 . The method of claim 1 , further comprising providing skip connections between the encoder portion and the decoder portion.
11 . The method of claim 1 , further comprising correcting the at least one encoded and/or spectrally sheared image for view differences comprising: using a machine learning algorithm which comprises a view correcting neural network, to determine a transform configured to map the first and second images to one another; and correcting for view differences by transforming one of the first image and the second image using the determined transform, wherein the method correcting for view differences comprises: concatenating the first image and the second image to form concatenated image data; providing the concatenated image data to the view correcting neural network; performing, using the view correcting neural network, one or more convolution operations; determining, using the view correcting neural network, the transform from an output of the one or more convolution operations; and transforming the one of the first image and the second image using the determined transform.
12 . The method of claim 1 , wherein the reconstructed cube comes from either an earlier layer of the network or an external source.
13 . The method of claim 1 , further comprising:
performing a plurality of convolution operations using the encoder portion; and
performing a plurality of transpose convolution operations using the decoder portion.
14 . The method of claim 1 , further comprising performing a plurality of successive dilated convolution operations using the encoder portion, wherein a dilation rate increases with each successive dilated convolution operation with an increase rate greater than 0.
15 . The method of claim 1 , wherein a plurality of successive convolution operations are densely connected.
16 . The method of claim 15 , wherein an output of each preceding densely connected convolution operation is provided as an input to a subsequent densely connected convolution operation.
17 . The method of claim 16 , wherein the outputs of each preceding densely connected convolution operation are concatenated to provide the input to the subsequent densely connected convolution operation.
18 . The method of claim 14 , wherein the plurality of successive dilated convolution operations appears within the encoder portion.
19 . The method of claim 1 , wherein the decoder portion comprises a self-attention block configured to receive a final output of the encoder portion.
20 . The method of claim 1 , further comprising performing spatially adaptive normalization on an output of at least one of the primary convolution operations and/or transpose convolution operations using the unencoded and spectrally unsheared image of the scene.
21 . The method of claim 1 , further comprising reconstructing the hyperspectral datacube using hierarchical channel reconstruction.
22 . The method of claim 1 , further comprising providing more estimated hyperspectral datacubes to the neural network.
23 . The method of claim 1 , wherein an output of the neural network is transformed by a relative normalization operator to its collapse and a resulting representation is remapped using one or more spectrally sheared or unsheared views.
24 . The method of claim 23 , wherein the relative normalization operator comprises a sigmoid operation which is applied to a reconstructed hypercube, and a result of the sigmoid operation is divided by a reduced sum of itself along its final axis.