Method, electronic device, and storage medium for obtaining depth image
View Patent ↗The present disclosure provides a method, an electronic device, and a storage medium for obtaining a depth image. The method includes: mapping a depth image having a first resolution into a color image having a second resolution to obtain a processed image, the first resolution being smaller than the second resolution; establishing an objective function with respect to the processed image and a depth image having the second resolution, pixels in the processed image having one-to-one correspondence with pixels in the depth image having the second resolution, the objective function aiming to minimize an error with respect to the processed image and the depth image having the second resolution; and solving the objective function to obtain the depth image having the second resolution.
1. A method for obtaining a depth image, comprising:
mapping a depth image having a first resolution into a color image having a second resolution to obtain a processed image, the first resolution being smaller than the second resolution;
establishing an objective function with respect to the processed image and a depth image having the second resolution, pixels in the processed image having one-to-one correspondence with pixels in the depth image having the second resolution, the objective function aiming to minimize an error with respect to the processed image and the depth image having the second resolution; and
solving the objective function to obtain the depth image having the second resolution;
wherein the processed image comprises a plurality of first pixels and a plurality of second pixels, each of the plurality of first pixels comprises color information and depth information, and each of the plurality of second pixels comprises color information,
wherein determining the objective function with respect to the processed image and the depth image having the second resolution comprises:
determining a data item of the objective function based on depth information of the plurality of first pixels in the processed image and depth information of a plurality of pixels corresponding to the plurality of first pixels in the depth image having the second resolution;
determining a neighborhood smoothing item of the objective function based on depth information of the plurality of first pixels in the processed image, and depth information of pixels in a first neighborhood corresponding to each of the plurality of first pixels in the depth image having the second resolution; and
determining a non-local average regularization item of the objective function based on depth information of the plurality of first pixels in the processed image, and depth information of pixels in a second neighborhood corresponding to each of the plurality of first pixels in the depth image having the second resolution,
the second neighborhood being greater than the first neighborhood.
2. The method of claim 1 , further comprising:
obtaining one or more low-confidence pixels at edges of the depth image having the first resolution based on a confidence map; and
removing depth information of the one or more low-confidence pixels.
3. The method of claim 2 , wherein the depth image having the first resolution is obtained by a time-of-flight camera component, the time-of-flight camera component is configured to sample repeatedly to obtain a plurality of phase maps, the depth image having the first resolution is obtained by processing the plurality of phase maps, and the method comprises:
obtaining an average pixel value of corresponding pixels in the plurality of phase maps;
obtaining an amplitude map based on a plurality of average pixel values; and
filtering the amplitude map to obtain the confidence map.
4. The method of claim 2 , wherein the depth image having the first resolution is obtained by a structured-light camera component, the structured-light camera component is configured to acquire a structured-light image, the depth image having the first resolution is obtained by processing the structured-light image, and the method comprises:
obtaining a maximum pixel value in the structured-light image; and
comparing each pixel value in the structured-light image with the maximum pixel value to obtain the confidence map.
5. The method of claim 1 , wherein the objective function is denoted by a formula of:
E ( D )= E d ( D )+λ s E s ( D )+λ N E NLM ( D ),
where, E(D) denotes the objective function, E d (D) denotes the data item, E s (D) denotes the neighborhood smoothing item, E NLM (D) denotes the non-local average regularization item, λ s and λ N denote weights.
6. The method of claim 5 , wherein the data item is denoted by a formula of:
E
d
(
D
)
=
∑
p
∈
P
(
D
(
p
)
-
G
(
p
)
)
2
,
where, p denotes a current pixel, P denotes a set of the plurality of first pixels, D(p) denotes depth information of the pixel p in the processed image, G(p) denotes depth information of a pixel corresponding to the pixel p in the depth image having the second resolution.
7. The method of claim 5 , wherein the neighborhood smoothing item is denoted by a formula of:
E
s
(
D
)
=
∑
p
∈
P
∑
q
∈
N
(
p
)
w
pq
(
D
(
p
)
-
D
(
q
)
)
2
,
where, p denotes a current pixel, P denotes a set of the plurality of first pixels, D(p) denotes depth information of the pixel p in the processed image, N(p) denotes a first neighborhood of a pixel corresponding to the pixel p in the depth image having the second resolution, q denotes a pixel in the first neighborhood, w pq denotes a first weight, and D (q) denotes depth information of the pixel q in the depth image having the second resolution.
8. The method of claim 7 , wherein the first weight is denoted by a formula of:
w pq =w c w s w e w d ,
where, w c denotes a color similarity, w s denotes a separation weight, w e denotes a salient edge weight, and w d denotes a higher-order linear interpolation weight;
the color similarity is denoted by a formula of
w
c
=
exp
-
(
∑
I
∈
YUV
(
I
(
p
)
-
I
(
q
)
)
2
2
σ
I
2
)
where I denotes luminance, chromaticity or density of YUV space, I(p) denotes color information of the pixel p in the processed image, I(q) denotes color information of the pixel q in the processed image, and σ I denotes a preset constant;
the separation weight is denoted by a formula of w s =t se , t se denotes a preset constant;
the salient edge weight is denoted by a formula of
w
e
=
1
s
x
(
p
)
2
+
s
x
(
q
)
2
+
1
,
where s x (p) denotes a pixel value corresponding to the pixel p in an image obtained by applying Sobel operator to the color image in an X axis, and s x (q) denotes a pixel value corresponding to the pixel q in an image obtained by applying Sobel operator to the color image in an X axis; and
the higher-order linear interpolation weight is denoted by a formula of
w
d
=
exp
-
(
∑
(
D
g
(
p
)
-
D
g
(
q
)
)
2
2
σ
g
2
)
,
where D g (p) denotes a pixel value corresponding to the pixel p in an image obtained by applying Gaussian interpolation to the depth image having the first resolution, D g (q) denotes a pixel value corresponding to the pixel q in an image obtained by applying Gaussian interpolation to the depth image having the second resolution, and σ g denotes a preset constant.
9. The method of claim 5 , wherein the non-local average regularization item is denoted by a formula of:
E
NLM
(
D
)
=
∑
p
∈
P
∑
r
∈
A
(
p
)
k
pr
(
D
(
p
)
-
D
(
r
)
)
2
,
where, p denotes a current pixel, P denotes a set of the plurality of first pixels, D(p) denotes depth information of the pixel p in the processed image, A(p) denotes a second neighborhood of a pixel corresponding to the pixel p in the depth image having the second resolution, r denotes a pixel in the second neighborhood, k pr denotes a second weight, and D(r) is depth information of the pixel r in the depth image having the second resolution.
10. The method of claim 1 , wherein solving the objective function to obtain the depth image having the second resolution comprises:
finding a minimum solution to the objective function with an optimization-problem solving algorithm to obtain the depth image having the second resolution.
11. An electronic device, comprising:
a memory configured to store instructions; and
a processor configured to read the instructions to carry out the following acts:
mapping a depth image having a first resolution into a color image having a second resolution to obtain a processed image, the first resolution being smaller than the second resolution;
establishing an objective function with respect to the processed image and a depth image having the second resolution, pixels in the processed image having one-to-one correspondence with pixels in the depth image having the second resolution, the objective function aiming to minimize an error with respect to the processed image and the depth image having the second resolution; and
solving the objective function to obtain the depth image having the second resolution;
wherein the processed image comprises a plurality of first pixels and a plurality of second pixels, each of the plurality of first pixels comprises color information and depth information, and each of the plurality of second pixels comprises color information,
wherein determining the objective function with respect to the processed image and the depth image having the second resolution comprises:
determining a data item of the objective function based on depth information of the plurality of first pixels in the processed image and depth information of a plurality of pixels corresponding to the plurality of first pixels in the depth image having the second resolution;
determining a neighborhood smoothing item of the objective function based on depth information of the plurality of first pixels in the processed image, and depth information of pixels in a first neighborhood corresponding to each of the plurality of first pixels in the depth image having the second resolution; and
determining a non-local average regularization item of the objective function based on depth information of the plurality of first pixels in the processed image, and depth information of pixels in a second neighborhood corresponding to each of the plurality of first pixels in the depth image having the second resolution,
the second neighborhood being greater than the first neighborhood.
12. The electronic device of claim 11 , wherein the processor is configured to carry out the following acts:
obtaining one or more low-confidence pixels at edges of the depth image having the first resolution based on a confidence map; and
removing depth information of the one or more low-confidence pixels.
13. The electronic device of claim 12 , wherein the depth image having the first resolution is obtained by a time-of-flight camera component, the time-of-flight camera component is configured to sample repeatedly to obtain a plurality of phase maps, the depth image having the first resolution is obtained by processing the plurality of phase maps, and the processor is configured to carry out the following acts:
obtaining an average pixel value of corresponding pixels in the plurality of phase maps;
obtaining an amplitude map based on a plurality of average pixel values; and
filtering the amplitude map to obtain the confidence map,
or
wherein the depth image having the first resolution is obtained by a structured-light camera component, the structured-light camera component is configured to acquire a structured-light image, the depth image having the first resolution is obtained by processing the structured-light image, and the processor is configured to carry out the following acts:
obtaining a maximum pixel value in the structured-light image; and
comparing each pixel value in the structured-light image with the maximum pixel value to obtain the confidence map.
14. The electronic device of claim 11 , wherein the objective function is denoted by a formula of:
E ( D )= E d ( D )+λ s E s ( D )+λ N E NLM ( D ),
where, E(D) denotes the objective function, E d (D) denotes the data item, E s (D) denotes the neighborhood smoothing item, E NLM (D) denotes the non-local average regularization item, λ s and λ N denote weights.
15. The electronic device of claim 14 , wherein the data item is denoted by a formula of:
E
d
(
D
)
=
∑
p
∈
P
(
D
(
p
)
-
G
(
p
)
)
2
,
where, p denotes a current pixel, P denotes a set of the plurality of first pixels, D(p) denotes depth information of the pixel p in the processed image, G(p) denotes depth information of a pixel corresponding to the pixel p in the depth image having the second resolution.
16. The electronic device of claim 14 , wherein the neighborhood smoothing item is denoted by a formula of:
E
s
(
D
)
=
∑
p
∈
P
∑
q
∈
N
(
p
)
w
pq
(
D
(
p
)
-
D
(
q
)
)
2
,
where, p denotes a current pixel, P denotes a set of the plurality of first pixels, D(p) denotes depth information of the pixel p in the processed image, N(p) denotes a first neighborhood of a pixel corresponding to the pixel p in the depth image having the second resolution, q denotes a pixel in the first neighborhood, w pq denotes a first weight, and D (q) denotes depth information of the pixel q in the depth image having the second resolution.
17. The electronic device of claim 14 , wherein the non-local average regularization item is denoted by a formula of:
E
NLM
(
D
)
=
∑
p
∈
P
∑
r
∈
A
(
p
)
k
pr
(
D
(
p
)
-
D
(
r
)
)
2
,
where, p denotes a current pixel, P denotes a set of the plurality of first pixels, D(p) denotes depth information of the pixel p in the processed image, A(p) denotes a second neighborhood of a pixel corresponding to the pixel p in the depth image having the second resolution, r denotes a pixel in the second neighborhood, k pr denotes a second weight, and D(r) is depth information of the pixel r in the depth image having the second resolution.
18. A non-transitory computer readable storage medium, comprising a computer program used in combination with an electronic device, the computer program being executable by a processor of the electronic device to implement:
mapping a depth image having a first resolution into a color image having a second resolution to obtain a processed image, the first resolution being smaller than the second resolution;
establishing an objective function with respect to the processed image and a depth image having the second resolution, pixels in the processed image having one-to-one correspondence with pixels in the depth image having the second resolution, the objective function aiming to minimize an error with respect to the processed image and the depth image having the second resolution; and
solving the objective function to obtain the depth image having the second resolution;
wherein the processed image comprises a plurality of first pixels and a plurality of second pixels, each of the plurality of first pixels comprises color information and depth information, and each of the plurality of second pixels comprises color information,
wherein determining the objective function with respect to the processed image and the depth image having the second resolution comprises:
determining a data item of the objective function based on depth information of the plurality of first pixels in the processed image and depth information of a plurality of pixels corresponding to the plurality of first pixels in the depth image having the second resolution;
determining a neighborhood smoothing item of the objective function based on depth information of the plurality of first pixels in the processed image, and depth information of pixels in a first neighborhood corresponding to each of the plurality of first pixels in the depth image having the second resolution; and
determining a non-local average regularization item of the objective function based on depth information of the plurality of first pixels in the processed image, and depth information of pixels in a second neighborhood corresponding to each of the plurality of first pixels in the depth image having the second resolution,
the second neighborhood being greater than the first neighborhood.
19. The non-transitory computer readable storage medium of claim 18 , wherein the objective function is denoted by a formula of:
E ( D )= E d ( D )+λ s E s ( D )+λ N E NLM ( D ),
where, E(D) denotes the objective function, E d (D) denotes the data item, E s (D) denotes the neighborhood smoothing item, E NLM (D) denotes the non-local average regularization item, λ s and λ N denote weights.
20. The non-transitory computer readable storage medium of claim 18 , wherein solving the objective function to obtain the depth image having the second resolution comprises:
finding a minimum solution to the objective function with an optimization-problem solving algorithm to obtain the depth image having the second resolution.