Granted Patent B2

US 12,657,435 · App. 18/341,617 · Granted Jun 16, 2026

Inventors: Qi Cao (Hangzhou, CN); Xiaoli Zhang (Hangzhou, CN); Di Zhang (Hangzhou, CN); Haihang Zhang (Shenzhen, CN)

Assignee: Yinwang Intelligent Technologies Co., Ltd.

G06N3/0455G06T5/60G06T7/136G06T7/50G06T7/90G06T2207/10024G06T2207/10028G06T2207/20084G06T2207/30192

Abstract

Claims (55)

1 . An image processing method, comprising:

obtaining, by an image processing apparatus, a first depth map and a color image, the first depth map and the color image being for a same scene;

determining, by the image processing apparatus, an atmospheric light value;

performing, by the image processing apparatus, depth completion on the first depth map to obtain a second depth map, wherein performing the depth completion on the first depth map to obtain the second depth map comprises:

providing the color image and a second intermediate feature image to a first deep neural network to obtain a third depth map, wherein the second intermediate feature image is generated at an intermediate layer of a second deep neural network;

providing the first depth map and a first intermediate feature image to the second deep neural network to obtain a fourth depth map, wherein the first intermediate feature image is generated at an intermediate layer of the first deep neural network; and

performing a fusion operation on the third depth map and the fourth depth map to obtain the second depth map;

determining, by the image processing apparatus, a transmittance based on the second depth map; and

performing, by the image processing apparatus, repair processing on the color image based on the atmospheric light value and the transmittance.

2 . The image processing method according to claim 1 , wherein the first deep neural network comprises a first preprocessing network, a first encoder, and a first decoder, wherein the first preprocessing network is configured to transform a color image that is input to the first deep neural network into a first feature image suitable for processing by the first encoder, wherein the first encoder is configured to perform feature encoding on the first feature image, and wherein the first decoder is configured to perform feature decoding on a second feature image that is output by the first encoder.

3 . The image processing method according to claim 2 , wherein the first intermediate feature image comprises a convolutional layer feature image generated at a convolutional layer in an encoding unit of the first encoder and an upsampling layer feature image generated at an upsampling layer in a decoding unit of the first decoder.

4 . The image processing method according to claim 1 , wherein the second deep neural network comprises a second preprocessing network, a second encoder, and a second decoder, wherein the second preprocessing network is configured to transform a first depth map that is input to the second deep neural network into a third feature image suitable for processing by the second encoder, wherein the second encoder is configured to perform feature encoding on the third feature image, and wherein the second decoder is configured to perform feature decoding on a fourth feature image that is output by the second encoder.

5 . The image processing method according to claim 4 , wherein the second intermediate feature image comprises a convolutional layer feature image generated at a convolutional layer in an encoding unit of the second encoder and an upsampling layer feature image generated at an upsampling layer in a decoding unit of the second decoder.

6 . The image processing method according to claim 1 , wherein determining the atmospheric light value is based on the first depth map.

7 . The image processing method according to claim 6 , wherein determining the atmospheric light value based on the first depth map comprises:

determining, from the first depth map, a sky depth map used to indicate a sky region; and

determining the atmospheric light value based on the sky depth map and the color image.

8 . The image processing method according to claim 7 , wherein determining the sky depth map uses a sliding window method.

9 . The image processing method according to claim 8 , wherein a window shape of a sliding window used in the sliding window method is a rectangle with a size of one row of pixels, and a step size of the sliding window is 1.

10 . The image processing method according to claim 7 , wherein determining the atmospheric light value based on the sky depth map and the color image comprises:

determining, from the color image using the sky depth map, a sky color image used to indicate the sky region; and

determining the atmospheric light value based on the sky color image.

11 . The image processing method according to claim 10 , wherein determining the sky color image comprises:

performing a binarization operation on the first depth map using the sky depth map to obtain a binary map; and

determining, based on the binary map and the color image, the sky color image.

12 . The image processing method according to claim 10 , wherein determining the atmospheric light value based on the sky color image comprises:

determining pixel points of a brightest part of the sky color image; and

determining, as the atmospheric light value, an average value of the pixel points of the brightest part of the sky color image.

13 . An image processing apparatus, comprising:

a memory storing instructions; and

at least one processor in communication with the memory, wherein the at least one processor is configured, upon execution of the instructions, to perform the following-steps:

obtain a first depth map and a color image, the first depth map and the color image being for a same scene;

determine an atmospheric light value;

perform depth completion on the first depth map to obtain a second depth map, wherein performing the depth completion on the first depth map to obtain the second depth map comprises:

providing the color image and a second intermediate feature image to a first deep neural network to obtain a third depth map, wherein the second intermediate feature image is generated at an intermediate layer of a second deep neural network;

providing the first depth map and a first intermediate feature image to the second deep neural network to obtain a fourth depth map, wherein the first intermediate feature image is generated at an intermediate layer of the first deep neural network; and

performing a fusion operation on the third depth map and the fourth depth map to obtain the second depth map;

determine a transmittance based on the second depth map; and

perform repair processing on the color image based on the atmospheric light value and the transmittance.

14 . The image processing apparatus according to claim 13 , wherein the first deep neural network comprises a first preprocessing network, a first encoder an a first decoder, wherein the first preprocessing network is configured to transform a color image that is input to the first deep neural network into a first feature image suitable for processing by the first encoder, wherein the first encoder is configured to perform feature encoding on the first feature image, and wherein the first decoder is configured to perform feature decoding on a second feature image that is output by the first encoder.

15 . The image processing apparatus according to claim 14 , wherein the first intermediate feature image comprises a convolutional layer feature image generated at a convolutional layer in an encoding unit of the first encoder and an upsampling layer feature image generated at an upsampling layer in a decoding unit of the first decoder.

16 . The image processing apparatus according to claim 13 , wherein the second deep neural network comprises a second preprocessing network, a second encoder, and a second decoder, wherein the second preprocessing network is configured to transform a first depth map that is input to the second deep neural network into a third feature image suitable for processing by the second encoder, wherein the second encoder is configured to perform feature encoding on the third feature image, and wherein the second decoder is configured to perform feature decoding on a fourth feature image that is output by the second encoder.

17 . The image processing apparatus according to claim 13 , wherein determining the atmospheric light value is based on the first depth map.

18 . A non-transitory computer-readable medium having processor-executable instructions stored thereon for image processing, wherein the processor-executable instructions, when executed, facilitate performance of the following:

obtaining a first depth map and a color image, the first depth map and the color image being for a same scene;

determining an atmospheric light value;

performing depth completion on the first depth map to obtain a second depth map, wherein performing the depth completion on the first depth map to obtain the second depth map comprises:

providing the color image and a second intermediate feature image to a first deep neural network to obtain a third depth map, wherein the second intermediate feature image is generated at an intermediate layer of a second deep neural network;

providing the first depth map and a first intermediate feature image to the second deep neural network to obtain a fourth depth map, wherein the first intermediate feature image is generated at an intermediate layer of the first deep neural network; and

performing a fusion operation on the third depth map and the fourth depth map to obtain the second depth map;

determining a transmittance based on the second depth map; and

performing repair processing on the color image based on the atmospheric light value and the transmittance.

19 . The non-transitory computer-readable medium according to claim 18 , wherein the first deep neural network comprises a first preprocessing network, a first encoder, and a first decoder, wherein the first preprocessing network is configured to transform a color image that is input to the first deep neural network into a first feature image suitable for processing by the first encoder, wherein the first encoder is configured to perform feature encoding on the first feature image, and wherein the first decoder is configured to perform feature decoding on a second feature image that is output by the first encoder; and

wherein the second deep neural network comprises a second preprocessing network, a second encoder, and a second decoder, wherein the second preprocessing network is configured to transform a first depth map that is input to the second deep neural network into a third feature image suitable for processing by the second encoder, wherein the second encoder is configured to perform feature encoding on the third feature image, and wherein the second decoder is configured to perform feature decoding on a fourth feature image that is output by the second encoder.

20 . The non-transitory computer-readable medium according to claim 18 , wherein determining the atmospheric light value is based on the first depth map.

Assignments (3)

Priority Claims (1)

Continuity (2)

References Cited (27)