Granted Patent B2

US 12,198,301 · App. 17/770,605 · Granted Jan 14, 2025

Inventor: Yanan Zhang (Hangzhou, CN)

Assignee: ZHEJIANG UNIVIEW TECHNOLOGIES CO., LTD.

G06T5/50G06T5/70G06T5/94G06T7/74H04N9/78G06T2207/10024G06T2207/10048G06T2207/20192G06T2207/20221

Abstract

Claims (88)

1. An image fusion method, comprising:

acquiring a visible light image and an infrared image which are to-be-fused;

performing luminance and chrominance separation on the visible light image to extract a luminance component of the visible light image and a chrominance component of the visible light image;

performing luminance fusion on the luminance component of the visible light image and the infrared image to obtain a luminance fusion result; and

performing image reconstruction according to the luminance fusion result and the chrominance component of the visible light image to obtain a fused image,

wherein performing the luminance fusion on the luminance component of the visible light image and the infrared image to obtain the luminance fusion result comprises:

correcting the infrared image according to the luminance component of the visible light image to obtain a corrected infrared image;

performing image layer decomposition on the luminance component of the visible light image and the corrected infrared image, respectively, and performing corresponding fusion on a plurality of layers of the luminance component of the visible light image obtained after the image layer decomposition and a plurality of layers of the corrected infrared image obtained after the image layer decomposition; and

superimposing results of performing corresponding fusion on the plurality of layers of the luminance component of the visible light image obtained after the image layer decomposition and the plurality of layers of the corrected infrared image obtained after the image layer decomposition to obtain the luminance fusion result.

2. The method according to claim 1 , wherein correcting the infrared image according to the luminance component of the visible light image to obtain the corrected infrared image comprises:

determining a position of a reference pixel in the luminance component of the visible light image according to a position of each pixel in the infrared image; and

determining a luminance correction result of the each pixel according to a preset-range neighborhood block with the position of the reference pixel as a center and a preset-range neighborhood block with the position of the each pixel as a center to obtain the corrected infrared image.

3. The method according to claim 2 , wherein determining the luminance correction result of the each pixel according to the preset-range neighborhood block with the position of the reference pixel as the center and the preset-range neighborhood block with the position of the each pixel as the center comprises:

determining the luminance correction result of the each pixel by adopting the following formula:

Y ir ′( i )= Y vis ( i )α i (1)+α i (2);

wherein Y ir ′(i) represents the luminance correction result of the each pixel, Y vis (i) represents a luminance value of the reference pixel, and α i (1) and α i (2) represent a first numerical value of a matrix α i and a second numerical value of the matrix α i ;

wherein α i =( Q i T W i Q i +λI ) −1 ( Q i T W i p i +λα i 0 ), α i ΠR 2×1 ;

wherein λ represents a preset regularization parameter, W i represents a preset weight matrix, and Q i represents a matrix formed by luminance values of a plurality of pixels within the preset-range neighborhood block with the position of the each pixel as the center and a numerical value 1; Q i T represents a transposed matrix of Q i ; p i represents a matrix formed by luminance values of pixels within the preset-range neighborhood block with the position of the reference pixel as the center; I represents an identity matrix; α i 0 represents a local contrast factor formed by a ratio of a luminance value of the each pixel to an average value of the luminance values of the plurality of pixels within the preset-range neighborhood block with the position of the each pixel as the center; and R 2×1 represents a linear space formed by all 2×1 matrices over a real number field R.

4. The method according to claim 1 , wherein performing the image layer decomposition on the luminance component of the visible light image and the corrected infrared image, respectively, and performing the corresponding fusion on the plurality of layers of the luminance component of the visible light image obtained after the image layer decomposition and on the plurality of layers of the corrected infrared image obtained after the image layer decomposition comprises:

performing decomposition to the luminance component of the visible light image into a visible light luminance base layer and a visible light luminance detail layer, and performing decomposition to the corrected infrared image into an infrared image base layer and an infrared image detail layer; and

fusing the visible light luminance base layer and the infrared image base layer, and fusing the visible light luminance detail layer and the infrared image detail layer.

5. The method according to claim 4 , wherein fusing the visible light luminance base layer and the infrared image base layer comprises:

determining a region saliency matrix of the visible light luminance base layer and a region saliency matrix of the infrared image base layer through high-pass filtering, and determining a first weight B vis 1 of the visible light luminance base layer and a first weight B ir 1 of the infrared image base layer according to the region saliency matrix of the visible light luminance base layer and the region saliency matrix of the infrared image base layer;

determining a second weight B vis 2 of the visible light luminance base layer and a second weight B ir 2 of the infrared image base layer according to a preset optimal luminance value;

determining a fusion weight of the visible light luminance base layer according to the first weight B vis 1 of the visible light luminance base layer and the second weight B vis 2 of the visible light luminance base layer; and determining a fusion weight of the infrared image base layer according to the first weight B ir 1 of the infrared image base layer and the second weight B ir 2 of the infrared image base layer; and

fusing the visible light luminance base layer and the infrared image base layer according to the fusion weight of the visible light luminance base layer and the fusion weight of the infrared image base layer.

6. The method according to claim 4 , wherein fusing the visible light luminance detail layer and the infrared image detail layer comprises:

calculating an edge strength matrix of the visible light luminance detail layer and an edge strength matrix of the infrared image detail layer, and determining a first weight D vis 1 of the visible light luminance detail layer and a first weight D ir 1 of the infrared image detail layer based on an edge strength matrix of the visible light luminance detail layer and an edge strength matrix of the infrared image detail layer;

determining a second weight D vis 2 of the visible light luminance detail layer and a second weight D ir 2 of the infrared image detail layer according to a preset optimal edge strength value;

determining a fusion weight of the visible light luminance detail layer according to the first weight D vis 1 of the visible light luminance detail layer and the second weight D vis 2 of the visible light luminance detail layer; and determining a fusion weight of the infrared image detail layer according to the first weight D ir 1 of the infrared image detail layer and the second weight D ir 2 of the infrared image detail layer; and

fusing the visible light luminance detail layer and the infrared image detail layer according to the fusion weight of the visible light luminance detail layer and the fusion weight of the infrared image detail layer.

7. A non-transitory computer-readable storage medium, storing a computer program, wherein the computer program, when executed by a processor, implements:

acquiring a visible light image and an infrared image which are to-be-fused;

performing luminance and chrominance separation on the visible light image to extract a luminance component of the visible light image and a chrominance component of the visible light image;

performing luminance fusion on the luminance component of the visible light image and the infrared image to obtain a luminance fusion result; and

performing image reconstruction according to the luminance fusion result and the chrominance component of the visible light image to obtain a fused image,

wherein performing the luminance fusion on the luminance component of the visible light image and the infrared image to obtain the luminance fusion result comprises:

correcting the infrared image according to the luminance component of the visible light image to obtain a corrected infrared image;

performing image layer decomposition on the luminance component of the visible light image and the corrected infrared image, respectively, and performing corresponding fusion on a plurality of layers of the luminance component of the visible light image obtained after the image layer decomposition and a plurality of layers of the corrected infrared image obtained after the image layer decomposition; and

superimposing results of performing corresponding fusion on the plurality of layers of the luminance component of the visible light image obtained after the image layer decomposition and the plurality of layers of the corrected infrared image obtained after the image layer decomposition to obtain the luminance fusion result.

8. The non-transitory computer-readable storage medium according to claim 7 , wherein the computer program, when executed by a processor, implements:

determining a position of a reference pixel in the luminance component of the visible light image according to a position of each pixel in the infrared image; and

determining a luminance correction result of the each pixel according to a preset-range neighborhood block with the position of the reference pixel as a center and a preset-range neighborhood block with the position of the each pixel as a center to obtain the corrected infrared image.

9. The non-transitory computer-readable storage medium according to claim 8 , wherein the computer program, when executed by a processor, implements:

determining the luminance correction result of the each pixel by adopting the following formula:

Y ir ′( i )= Y vis ( i )α i (1)+α i (2);

wherein Y ir ′(i) represents the luminance correction result of the each pixel, Y vis (i) represents a luminance value of the reference pixel, and α i (1) and α i (2) represent a first numerical value of a matrix α i and a second numerical value of the matrix α i ;

wherein α i =( Q i T W i Q i +λI ) −1 ( Q i T W i p i +λα i 0 ), α i ΠR 2×1 ;

wherein λ represents a preset regularization parameter, W i represents a preset weight matrix, and Q i represents a matrix formed by luminance values of a plurality of pixels within the preset-range neighborhood block with the position of the each pixel as the center and a numerical value 1; Q i T represents a transposed matrix of Q i ; p i represents a matrix formed by luminance values of pixels within the preset-range neighborhood block with the position of the reference pixel as the center; I represents an identity matrix; α i 0 represents a local contrast factor formed by a ratio of a luminance value of the each pixel to an average value of the luminance values of the plurality of pixels within the preset-range neighborhood block with the position of the each pixel as the center; and R 2×1 represents a linear space formed by all 2×1 matrices over a real number field R.

10. The non-transitory computer-readable storage medium according to claim 7 , wherein the computer program, when executed by a processor, implements:

performing decomposition to the luminance component of the visible light image into a visible light luminance base layer and a visible light luminance detail layer, and performing decomposition to the corrected infrared image into an infrared image base layer and an infrared image detail layer; and

fusing the visible light luminance base layer and the infrared image base layer, and fusing the visible light luminance detail layer and the infrared image detail layer.

11. The non-transitory computer-readable storage medium according to claim 10 , wherein the computer program, when executed by a processor, implements:

determining a region saliency matrix of the visible light luminance base layer and a region saliency matrix of the infrared image base layer through high-pass filtering, and determining a first weight B vis 1 of the visible light luminance base layer and a first weight B ir 1 of the infrared image base layer according to the region saliency matrix of the visible light luminance base layer and the region saliency matrix of the infrared image base layer;

determining a second weight B vis 2 of the visible light luminance base layer and a second weight B ir 2 of the infrared image base layer according to a preset optimal luminance value;

determining a fusion weight of the visible light luminance base layer according to the first weight B vis 1 of the visible light luminance base layer and the second weight B vis 2 of the visible light luminance base layer; and determining a fusion weight of the infrared image base layer according to the first weight B ir 1 of the infrared image base layer and the second weight B ir 2 of the infrared image base layer; and

fusing the visible light luminance base layer and the infrared image base layer according to the fusion weight of the visible light luminance base layer and the fusion weight of the infrared image base layer.

12. An electronic device, comprising a memory, a processor and a computer program stored in the memory and executable by the processor, wherein the processor, when executing the computer program, implements:

acquiring a visible light image and an infrared image which are to-be-fused;

performing luminance and chrominance separation on the visible light image to extract a luminance component of the visible light image and a chrominance component of the visible light image;

performing luminance fusion on the luminance component of the visible light image and the infrared image to obtain a luminance fusion result; and

performing image reconstruction according to the luminance fusion result and the chrominance component of the visible light image to obtain a fused image,

wherein performing the luminance fusion on the luminance component of the visible light image and the infrared image to obtain the luminance fusion result comprises:

correcting the infrared image according to the luminance component of the visible light image to obtain a corrected infrared image;

performing image layer decomposition on the luminance component of the visible light image and the corrected infrared image, respectively, and performing corresponding fusion on a plurality of layers of the luminance component of the visible light image obtained after the image layer decomposition and a plurality of layers of the corrected infrared image obtained after the image layer decomposition; and

superimposing results of performing corresponding fusion on the plurality of layers of the luminance component of the visible light image obtained after the image layer decomposition and the plurality of layers of the corrected infrared image obtained after the image layer decomposition to obtain the luminance fusion result.

13. The electronic device according to claim 12 , wherein the processor, when executing the computer program, implements:

determining a position of a reference pixel in the luminance component of the visible light image according to a position of each pixel in the infrared image; and

determining a luminance correction result of the each pixel according to a preset-range neighborhood block with the position of the reference pixel as a center and a preset-range neighborhood block with the position of the each pixel as a center to obtain the corrected infrared image.

14. The electronic device according to claim 13 , wherein the processor, when executing the computer program, implements:

determining the luminance correction result of the each pixel by adopting the following formula:

Y ir ′( i )= Y vis ( i )α i (1)+α i (2);

wherein Y ir ′(i) represents the luminance correction result of the each pixel, Y vis (i) represents a luminance value of the reference pixel, and α i (1) and α i (2) represent a first numerical value of a matrix α i and a second numerical value of the matrix α i ;

wherein α i =( Q i T W i Q i +λI ) −1 ( Q i T W i p i +λα i 0 ), α i ΠR 2×1 ;

wherein λ represents a preset regularization parameter, W i represents a preset weight matrix, and Q i represents a matrix formed by luminance values of a plurality of pixels within the preset-range neighborhood block with the position of the each pixel as the center and a numerical value 1; Q i T represents a transposed matrix of Q i ; p i represents a matrix formed by luminance values of pixels within the preset-range neighborhood block with the position of the reference pixel as the center; I represents an identity matrix; α i 0 represents a local contrast factor formed by a ratio of a luminance value of the each pixel to an average value of the luminance values of the plurality of pixels within the preset-range neighborhood block with the position of the each pixel as the center; and R 2×1 represents a linear space formed by all 2×1 matrices over a real number field R.

15. The electronic device according to claim 12 , wherein the processor, when executing the computer program, implements:

performing decomposition to the luminance component of the visible light image into a visible light luminance base layer and a visible light luminance detail layer, and performing decomposition to the corrected infrared image into an infrared image base layer and an infrared image detail layer; and

fusing the visible light luminance base layer and the infrared image base layer, and fusing the visible light luminance detail layer and the infrared image detail layer.

16. The electronic device according to claim 15 , wherein the processor, when executing the computer program, implements:

determining a region saliency matrix of the visible light luminance base layer and a region saliency matrix of the infrared image base layer through high-pass filtering, and determining a first weight B vis 1 of the visible light luminance base layer and a first weight B ir 1 of the infrared image base layer according to the region saliency matrix of the visible light luminance base layer and the region saliency matrix of the infrared image base layer;

determining a second weight B vis 2 of the visible light luminance base layer and a second weight B ir 2 of the infrared image base layer according to a preset optimal luminance value;

determining a fusion weight of the visible light luminance base layer according to the first weight B vis 1 of the visible light luminance base layer and the second weight B vis 2 of the visible light luminance base layer; and determining a fusion weight of the infrared image base layer according to the first weight B ir 1 of the infrared image base layer and the second weight B ir 2 of the infrared image base layer; and

fusing the visible light luminance base layer and the infrared image base layer according to the fusion weight of the visible light luminance base layer and the fusion weight of the infrared image base layer.

17. The electronic device according to claim 15 , wherein the processor, when executing the computer program, implements:

calculating an edge strength matrix of the visible light luminance detail layer and an edge strength matrix of the infrared image detail layer, and determining a first weight D vis 1 of the visible light luminance detail layer and a first weight D ir 1 of the infrared image detail layer based on an edge strength matrix of the visible light luminance detail layer and an edge strength matrix of the infrared image detail layer;

determining a second weight D vis 2 of the visible light luminance detail layer and a second weight D ir 2 of the infrared image detail layer according to a preset optimal edge strength value;

determining a fusion weight of the visible light luminance detail layer according to the first weight D vis 1 of the visible light luminance detail layer and the second weight D vis 2 of the visible light luminance detail layer; and determining a fusion weight of the infrared image detail layer according to the first weight D ir 1 of the infrared image detail layer and the second weight D ir 2 of the infrared image detail layer; and

fusing the visible light luminance detail layer and the infrared image detail layer according to the fusion weight of the visible light luminance detail layer and the fusion weight of the infrared image detail layer.

Priority Claims (1)

Continuity (1)

References Cited (20)

Cited By (2)

US 12,418,724 US 12,675,848