High-resolution hyperspectral computational imaging method and system and medium
The present invention discloses a high-resolution hyperspectral computational imaging method and system and a medium. The method of the present invention comprises: conducting spectral upsampling on an input RGB image Y to obtain an initial hyperspectral image X 0 ; and inputting the initial hyperspectral image X 0 into a pre-trained deep convolutional neural network guided by an imaging model, and conducting iteration computation to obtain a hyperspectral image X. The present invention can effectively achieve reconstruction of the RGB image to the high-resolution hyperspectral image and has the advantages of high reconstruction precision, high computational efficiency, little memory consumption and strong generalization ability.
1. A high-resolution hyperspectral computational imaging method, comprising:
1) conducting spectral upsampling on an input RGB image Y to obtain an initial hyperspectral image X 0 ; and
2) inputting the initial hyperspectral image X 0 into a pre-trained deep convolutional neural network guided by an imaging model, and conducting iteration computation to obtain a hyperspectral image X, wherein in the step 2), the deep convolutional neural network guided by an imaging model is formed by a plurality of modules of a same structure; the plurality of modules are in cascade connection; input of each module comprises the initial hyperspectral image X 0 and an upsampling result of a previous module or the initial hyperspectral image X 0 ; and the hyperspectral image X is obtained from output of the last module,
wherein each module comprises a hyperspectral prior learning (HPL) module and an imaging model guidance (IMG) module, the HPL module is used for learning prior features of the upsampling result of the previous module or the initial hyperspectral image X0, and the IMG module is used for optimizing the learned features based on the input initial hyperspectral image X0 and the prior features output by the HPL module according to the imaging model,
the HPL module is of a five-layer structure formed by sequentially connecting a 3×3 first convolutional layer, a nonlinear rectified linear unit, a channel attention mechanism, a 3×3 second convolutional layer and a spatial attention mechanism, each channel attention mechanism comprises a 1×1 convolutional operation, a nonlinear normalization unit, a linear arithmetic operation and a plurality of rectified linear units, and the 1×1 convolutional operation, the nonlinear normalization unit, the linear arithmetic operation and the plurality of rectified linear units are sequentially connected.
2. The high-resolution hyperspectral computational imaging method according to claim 1 , wherein in the step 1), a function expression of the initial hyperspectral image X 0 obtained by spectral upsampling on the input RGB image Y is:
X 0 =R † *Y (1)
in the above expression, R † is a generalized inverse of a spectral response function R.
3. The high-resolution hyperspectral computational imaging method according to claim 1 , wherein in the step 2), the step of conducting iteration computation to obtain the hyperspectral image X comprises:
2.1) initializing a number of iterations to be 1 and values of parameters in the deep convolutional neural network guided by the imaging model and a penalty factor μ t of a t th iteration;
2.2) learning the prior features of the upsampling result of the previous module or the initial hyperspectral image X 0 by the hyperspectral prior learning (HPL) module in the t th module firstly, optimizing the learned features based on the input initial hyperspectral image X 0 and the prior features output by the HPL module according to the imaging model by the IMG module, and updating the hyperspectral image X obtained by the t th iteration;
2.3) judging whether the condition that the number t of iterations is equal to a preset threshold T is satisfied or not; if the condition is satisfied, outputting the hyperspectral image X obtained by the t th iteration as a final result; or otherwise, adding 1 to the number t of iterations, and jumping to execute the step 2.2) for continuous iteration.
4. The high-resolution hyperspectral computational imaging method according to claim 1 , wherein the HPL module learns the prior features of the upsampling result of the previous module or the initial hyperspectral image X 0 , through the following steps:
a variable G is introduced and is updated by executing the expressions (3)-(4) to learn the upsampling result of the previous module or the initial hyperspectral image X 0 ,
G t+1 =argmin G λø( G )+μ t ∥G−X t ∥ F 2 (2)
X t+1 =argmin X ∥Y−FX∥ F 2 +μ t ∥G t+1 −X∥ F 2 (3)
μ t+1 =γμ t (4)
in the expressions, G t+1 is a value of the introduced variable in t+1th iteration; G is the introduced variable; λ is a weighting parameter; Ø (G) is a regularization term of the introduced variable G; μ t is a penalty factor of t th iteration; μ t+1 is a penalty factor of t+1 th iteration; X t is a hyperspectral image obtained by the t th iteration; X t+1 is a hyperspectral image obtained by the t+1 th iteration; Y represents an RGB image; F is a spectral response function; X represents a hyperspectral image; and γ is an update coefficient of the penalty factor.
5. The high-resolution hyperspectral computational imaging method according to claim 4 , wherein a function expression that each HPL module is used for optimizing the learned features based on the input initial hyperspectral image X 0 and the prior features output by the HPL module according to the imaging model is:
X t+1 =( F T F+μ t I ) −1 ( F T Y+μ t G t+1 ) (5)
in the expression, X t+1 is a result obtained by optimizing the learned features according to the imaging model; F is the spectral response function; I is a unit matrix, Y is the input RGB image; μ t is a penalty factor in the t th iteration; and G t+1 is a value of the introduced variable in the t+1 th iteration.
6. A high-resolution hyperspectral computational imaging system, comprising a microprocessor and a memory which are connected to each other, wherein the microprocessor is programmed or configured to execute the steps of the high-resolution hyperspectral computational imaging method of claim 1 .
7. A non-transitory computer readable storage medium, wherein computer programs programmed or configured to execute the high-resolution hyperspectral computational imaging method of claim 1 is stored in the computer readable storage medium.
8. A high-resolution hyperspectral computational imaging method, comprising:
1) Conducting spectral upsampling on an input RGB image Y to obtain an initial hyperspectral image X 0 ; and
2) Inputting the initial hyperspectral image X 0 into a pre-trained deep convolutional neural network guided by an imaging model, and conducting iteration computation to obtain a hyperspectral image X, wherein in the step 2), the deep convolutional neural network guided by an imaging model is formed by a plurality of modules of a same structure; the plurality of modules are in cascade connection; input of each module comprises the initial hyperspectral image X 0 and an upsampling result of a previous module or the initial hyperspectral image X 0 ;
and the hyperspectral image X is obtained from output of the last module, wherein each module comprises a hyperspectral prior learning (HPL) module and an imaging model guidance (IMG) module, the HPL module is used for learning prior features of the upsampling result of the previous module or the initial hyperspectral image X 0 , and the IMG module is used for optimizing the learned features based on the input initial hyperspectral image X 0 and the prior features output by the HPL module according to the imaging model, wherein in the step 2), the step of conducting iteration computation to obtain the hyperspectral image X comprises:
2.1) initializing a number of iterations to be 1 and values of parameters in the deep convolutional neural network guided by the imaging model and a penalty factor μ t of a t th iteration;
2.2) learning the prior features of the upsampling result of the previous module or the initial hyperspectral image X 0 by the HPL module in the t th module firstly, optimizing the learned features based on the input initial hyperspectral image X 0 and the prior features output by the HPL module according to the imaging model by the IMG module, and updating the hyperspectral image X obtained by the t th iteration;
2.3) judging whether the condition that the number t of iterations is equal to a preset threshold T is satisfied or not; if the condition is satisfied, outputting the hyperspectral image X obtained by the t th iteration as a final result; or otherwise, adding 1 to the number t of iterations, and jumping to execute the step 2.2) for continuous iteration.
9. The high-resolution hyperspectral computational imaging method according to claim 8 , wherein in the step 1), a function expression of the initial hyperspectral image X 0 obtained by spectral upsampling on the input RGB image Y is:
X 0 =R † *Y (1)
in the above expression, R † is a generalized inverse of a spectral response function R.
10. The high-resolution hyperspectral computational imaging method according to claim 8 , wherein the HPL module is of a five-layer structure formed by sequentially connecting a 3×3 first convolutional layer, a nonlinear rectified linear unit, a channel attention mechanism, a 3×3 second convolutional layer and a spatial attention mechanism, each channel attention mechanism comprises a 1×1 convolutional operation, a nonlinear normalization unit, a linear arithmetic operation and a plurality of rectified linear units, and the 1×1 convolutional operation, the nonlinear normalization unit, the linear arithmetic operation and the plurality of rectified linear units are sequentially connected.
11. The high-resolution hyperspectral computational imaging method according to claim 10 , wherein the HPL module learns the prior features of the upsampling result of the previous module or the initial hyperspectral image X 0 through the following steps:
a variable G is introduced and is updated by executing the expressions (3)-(4) to learn the upsampling result of the previous module or the initial hyperspectral image X 0 ,
G t+1 =argmin G λø( G )+μ t ∥G−X t ∥ F 2 (2)
X t+1 =argmin X ∥Y−FX∥ F 2 +μ t ∥G t+1 −X∥ F 2 (3)
μ t+1 =γμ t (4)
in the expressions, G t+1 is a value of the introduced variable in t+1 th iteration; G is the introduced variable; λ is a weighting parameter; Ø (G) is a regularization term of the introduced variable G; μ t is a penalty factor of t th iteration; μ t+1 is a penalty factor of t+1 th iteration; X t is a hyperspectral image obtained by the t th iteration; X t+1 is a hyperspectral image obtained by the t+1 th iteration; Y represents an RGB image; F is a spectral response function; X represents a hyperspectral image; and γ is an update coefficient of the penalty factor.
12. The high-resolution hyperspectral computational imaging method according to claim 11 , wherein a function expression that each HPL module is used for optimizing the learned features based on the input initial hyperspectral image X 0 and the prior features output by the hyperspectral prior learning (HPL) module according to the imaging model is:
X t+1 =( F T F+μ t I ) −1 ( F T Y+μ t G t+1 ) (5)
in the expression, X t+1 is a result obtained by optimizing the learned features according to the imaging model; F is the spectral response function; I is a unit matrix, Y is the input RGB image; μ t is a penalty factor in the t th iteration; and G t+1 is a value of the introduced variable in the t+1 th iteration.
13. A high-resolution hyperspectral computational imaging system, comprising a microprocessor and a memory which are connected to each other, wherein the microprocessor is programmed or configured to execute the steps of the high-resolution hyperspectral computational imaging method of claim 8 .
14. A non-transitory computer readable storage medium, wherein computer programs programmed or configured to execute the high-resolution hyperspectral computational imaging method of claim 8 is stored in the computer readable storage medium.
15. A high-resolution hyperspectral computational imaging method, comprising:
1) Conducting spectral upsampling on an input RGB image Y to obtain an initial hyperspectral image X 0 ; and
2) Inputting the initial hyperspectral image X 0 into a pre-trained deep convolutional neural network guided by an imaging model, and conducting iteration computation to obtain a hyperspectral image X, wherein in the step 2), the deep convolutional neural network guided by an imaging model is formed by a plurality of modules of a same structure; the plurality of modules are in cascade connection; input of each module comprises the initial hyperspectral image X 0 and an upsampling result of a previous module or the initial hyperspectral image X 0 ; and the hyperspectral image X is obtained from output of the last module,
wherein each module comprises a hyperspectral prior learning (HPL) module and an imaging model guidance (IMG) module, the HPL module is used for learning prior features of the upsampling result of the previous module or the initial hyperspectral image X0, and the IMG module is used for optimizing the learned features based on the input initial hyperspectral image X0 and the prior features output by the HPL module according to the imaging model,
wherein the HPL module learns the prior features of the upsampling result of the previous module or the initial hyperspectral image X 0 through the following steps:
a variable G is introduced and is updated by executing the expressions (3)-(4) to learn the upsampling result of the previous module or the initial hyperspectral image X 0 ,
G t+1 =argmin G λø( G )+μ t ∥G−X t ∥ F 2 (2)
X t+1 =argmin X ∥Y−FX∥ F 2 +μ t ∥G t+1 −X∥ F 2 (3)
μ t+1 =γμ t (4)
in the expressions, G t+1 is a value of the introduced variable in t+1 th iteration; G is the introduced variable; λ is a weighting parameter; ø (G) is a regularization term of the introduced variable G; μ t is a penalty factor of t th iteration; μ t+1 is a penalty factor of t+1 th iteration; X t is a hyperspectral image obtained by the t th iteration; X t+1 is a hyperspectral image obtained by the t+1 th iteration; Y represents an RGB image; F is a spectral response function; X represents a hyperspectral image; and γ is an update coefficient of the penalty factor, and
wherein a function expression that each HPL module is used for optimizing the learned features based on the input initial hyperspectral image X 0 and the prior features output by the hyperspectral prior learning (HPL) module according to the imaging model is:
X t+1 =( F T F+μ t I ) −1 ( F T Y+μ t G t+1 ) (5)
in the expression, X t+1 is a result obtained by optimizing the learned features according to the imaging model; F is the spectral response function; I is a unit matrix, Y is the input RGB image; μ t is a penalty factor in the t th iteration; and G t+1 is a value of the introduced variable in the t+1 th iteration.
16. The high-resolution hyperspectral computational imaging method according to claim 15 , wherein in the step 1), a function expression of the initial hyperspectral image X 0 obtained by spectral upsampling on the input RGB image Y is:
X 0 =R † *Y (1)
in the above expression, R † is a generalized inverse of a spectral response function R.
17. The high-resolution hyperspectral computational imaging method according to claim 15 , wherein the HPL module is of a five-layer structure formed by sequentially connecting a 3×3 first convolutional layer, a nonlinear rectified linear unit, a channel attention mechanism, a 3×3 second convolutional layer and a spatial attention mechanism, each channel attention mechanism comprises a 1×1 convolutional operation, a nonlinear normalization unit, a linear arithmetic operation and a plurality of rectified linear units, and the 1×1 convolutional operation, the nonlinear normalization unit, the linear arithmetic operation and the plurality of rectified linear units are sequentially connected.
18. The high-resolution hyperspectral computational imaging method according to claim 15 , wherein in the step 2), the step of conducting iteration computation to obtain the hyperspectral image X comprises:
2.1) initializing a number of iterations to be 1 and values of parameters in the deep convolutional neural network guided by the imaging model and a penalty factor μr of a t th iteration;
2.2) learning the prior features of the upsampling result of the previous module or the initial hyperspectral image X 0 by the HPL module in the t th module firstly, optimizing the learned features based on the input initial hyperspectral image X 0 and the prior features output by the HPL module according to the imaging model by the IMG module, and updating the hyperspectral image X obtained by the t th iteration;
2.3) judging whether the condition that the number t of iterations is equal to a preset threshold T is satisfied or not; if the condition is satisfied, outputting the hyperspectral image X obtained by the t th iteration as a final result; or otherwise, adding 1 to the number t of iterations, and jumping to execute the step 2.2) for continuous iteration.
19. A high-resolution hyperspectral computational imaging system, comprising a microprocessor and a memory which are connected to each other, wherein the microprocessor is programmed or configured to execute the steps of the high-resolution hyperspectral computational imaging method of claim 5 .
20. A non-transitory computer readable storage medium, wherein computer programs programmed or configured to execute the high-resolution hyperspectral computational imaging method of claim 15 is stored in the computer readable storage medium.