Depth information determining method and related apparatus

1. A depth information determining method for dual cameras, the method comprising:

obtaining, by processing circuitry, a t th left eye matching similarity from a left eye image captured by a first camera of the dual cameras to a right eye image captured by a second camera of the dual cameras, and a t th right eye matching similarity from the right eye image to the left eye image, t being an integer greater than 1;

processing, by the processing circuitry, the t th left eye matching similarity and a (t−1) th left eye attention map with a neural network model, to obtain a t th left eye disparity map;

processing, by the processing circuitry, the t th right eye matching similarity and a (t−1) th right eye attention map with the neural network model, to obtain a t th right eye disparity map;

determining, by the processing circuitry, first depth information according to the t th left eye disparity map; and

determining, by the processing circuitry, second depth information according to the t th right eye disparity map.

2. The depth information determining method according to claim 1 , further comprising:

mapping the t th right eye disparity map to a left eye coordinate system, to obtain a t th left eye mapping disparity map;

generating a t th left eye attention map according to the t th left eye mapping disparity map and the t th left eye disparity map;

mapping the t th left eye disparity map to a right eye coordinate system, to obtain a t th right eye mapping disparity map; and

generating a t th right eye attention map according to the t th right eye mapping disparity map and the t th right eye disparity map.

3. The depth information determining method according to claim 2 , wherein after the determining the first depth information and the determining the second depth information, the method further comprises:

obtaining a (t+1) th left eye matching similarity from the left eye image to the right eye image, and a (t+1) th right eye matching similarity from the right eye image to the left eye image;

processing the (t+1) th left eye matching similarity and the t th left eye attention map with the neural network model, to obtain a (t+1) th left eye disparity map;

processing the (t+1) th right eye matching similarity and the t th right eye attention map with the neural network model, to obtain a (t+1) th right eye disparity map;

determining third depth information according to the (t+1) th left eye disparity map; and

determining fourth depth information according to the (t+1) th right eye disparity map.

4. The depth information determining method according to claim 1 , wherein

the processing the t th left eye matching similarity and the (t−1) th left eye attention map includes:

obtaining a t th left eye hidden variable through calculation according to the t th left eye matching similarity and the (t−1) th left eye attention map by using a convolutional long short-term memory (ConvLSTM) network;

obtaining a t th left eye disparity cost according to the t th left eye hidden variable; and

calculating a t th predicted left eye disparity value according to the t th left eye disparity cost, the t th predicted left eye disparity value being used for generating the t th left eye disparity map; and

the processing the t th right eye matching similarity and the (t−1) th right eye attention map includes:

obtaining a t th right eye hidden variable through calculation according to the t th right eye matching similarity and the (t−1) th right eye attention map by using the ConvLSTM network;

obtaining a t th right eye disparity cost according to the t th right eye hidden variable; and

calculating a t th predicted right eye disparity value according to the t th right eye disparity cost, the t th predicted right eye disparity value being used for generating the t th right eye disparity map.

5. The depth information determining method according to claim 4 , wherein

the obtaining the t th left eye hidden variable includes calculating the t th left eye hidden variable based on

i′ t =σ( W xi *X′ t +W hi *H′ t−1 +W ci ∘ C′ t−1 ±b i ),

f′ t =σ( W xf *X′ t +W hf *H′ t−1 +W cf ∘ C′ t−1 +b f ),

o′ t =σ( W xo *X′ t +W ho *H′ t−1 +W co ∘ C′ t−1 ±b o ),

C′ t =f′ t ∘ C′ t−1 +t′ t ∘ tanh( W xc *X′ t +W hc *H′ t−1 ±b c ), and

H′ t =o′ t ∘ tanh( C′ t );

i′ t represents a network input gate of a t th left eye recursion, * represents multiplication of vectors, ∘ represents a convolution operation, σ represents a sigmoid function, W xi , W hi , W ci , and b i represent model parameters of the network input gate, X′ t represents the t th left eye matching similarity and the (t−1) th left eye attention map, f′ t represents a forget gate of the t th left eye recursion, W xf , W hf , W cf ; and b f represent model parameters of the forget gate, o′ t represents an output gate of the t th left eye recursion, W xo , W ho , W co , and b o represent model parameters of the output gate, C′ t represents a memory cell of the t th left eye recursion, C′ t−1 represents a memory cell of a (t−1) th left eye recursion, tan h represents a hyperbolic tangent function, H′ t−1 represents a (t−1) th left eye hidden variable, and H′ t represents the t th left eye hidden variable;

the obtaining a t th right eye hidden variable includes calculating the t th right eye hidden variable based on

i″ t =σ( W xi *X″ t +W hi *H′ t−1 +W ci ∘ C″ t−1 +b i ),

f″ t =σ( W xf *X″ t +W hf *H″ t−1 +W cf ∘ C″ t−1 +b f ),

o″ t =σ( W xo *X″ t +W ho *H″ t−1 +W co ∘ C″ t−1 +b o ),

C″ t =f t ∘ C″ t−1 +i′ t ∘ tanh( W xc *X″ t +W hc *H″ t−1 +b c ), and

H″ t =o″ t ∘ tanh( C″ t ), where

i″ t represents a network input gate of a t th right eye recursion, X″ t represents the t th right eye matching similarity and the (t−1) th right eye attention map, f″ t represents a forget gate of the t th right eye recursion, o″ t represents an output gate of the t th right eye recursion, C″ t represents a memory cell of the t th right eye recursion, C″ t−1 represents a memory cell of a (t−1) th right eye recursion, H″ t−1 represents a (t−1) th right eye hidden variable, and H″ t represents the t th right eye hidden variable.

6. The depth information determining method according to claim 4 , wherein

the obtaining the t th left eye disparity cost includes processing the t th left eye hidden variable with at least two fully connected layers, to obtain the t th left eye disparity cost; and

the obtaining the t th right eye disparity cost includes processing the t th right eye hidden variable with the at least two fully connected layers, to obtain the t th right eye disparity cost.

7. The depth information determining method according to claim 4 , wherein

the calculating the t th predicted left eye disparity value according to the t th left eye disparity cost includes calculating the t th predicted left eye disparity value based on d′*=Σ d=1 Dmax d′*σ(−c′ d );

d′* represents the t th predicted left eye disparity value, D max represents a maximum quantity in different disparity maps, d′ represents a t th left eye disparity value, σ represents a sigmoid function, and c′ d represents the t th left eye disparity cost;

the calculating the t th predicted right eye disparity value includes calculating the t th predicted right eye disparity value based on d″*=Σ d=1 Dmax d″*σ(−c″ d ); and

d″* represents the t th predicted right eye disparity value, c″ d represents the t th right eye disparity cost, and d″ represents a t th right eye disparity value.

8. The depth information determining method according to claim 5 , wherein

the determining the first depth information includes calculating the first depth information based on

Z

′

=

B

⁢

f

d

′

*

;

Z′ represents the first depth information, d′* represents the t th predicted left eye disparity value, B represents a binocular camera spacing, and f represents a focal length;

the determining the second depth information includes calculating the second depth information based on

Z

″

=

B

⁢

f

d

″

*

;

and

Z″ represents the second depth information, and d″ represents the t th predicted right eye disparity value.

9. A depth information determining apparatus, comprising:

processing circuitry configured to

obtain a t th left eye matching similarity from a left eye image captured by a first camera of dual cameras to a right eye image captured by a second camera of the dual cameras, and a t th right eye matching similarity from the right eye image to the left eye image, t being an integer greater than 1;

process the t th left eye matching similarity and a (t−1) th left eye attention map with a neural network model, to obtain a t th left eye disparity map;

process the t th right eye matching similarity and a (t−1) th right eye attention map with the neural network model, to obtain a t th right eye disparity map;

determine first depth information according to the t th left eye disparity map; and

determine second depth information according to the t th right eye disparity map.

10. The depth information determining apparatus according to claim 9 , wherein processing circuitry is configured to

map the t th right eye disparity map to a left eye coordinate system, to obtain a t th left eye mapping disparity map;

generate a t th left eye attention map according to the t th left eye mapping disparity map and the t th left eye disparity map;

map the t th left eye disparity map to a right eye coordinate system, to obtain a t th right eye mapping disparity map; and

generate a t th right eye attention map according to the t th right eye mapping disparity map and the t th right eye disparity map.

11. The depth information determining apparatus according to claim 10 , wherein the processing circuitry is configured to

obtain a (t+1) th left eye matching similarity from the left eye image to the right eye image, and a (t+1) th right eye matching similarity from the right eye image to the left eye image;

process the (t+1) th left eye matching similarity and the t th left eye attention map with the neural network model, to obtain a (t+1) th left eye disparity map;

process the (t+1) th right eye matching similarity and the t th right eye attention map with the neural network model, to obtain a (t+1) th right eye disparity map;

determine third depth information according to the (t+1) th left eye disparity map; and

determine fourth depth information according to the (t+1) th right eye disparity map.

12. The depth information determining apparatus according to claim 9 , wherein the processing circuitry is configured to

obtain a t th left eye hidden variable through calculation according to the t th left eye matching similarity and the (t−1) th left eye attention map by using a convolutional long short-term memory (ConvLSTM) network;

obtain a t th left eye disparity cost according to the t th left eye hidden variable;

calculate a t th predicted left eye disparity value according to the t th left eye disparity cost, the t th predicted left eye disparity value being used for generating the t th left eye disparity map;

obtain a t th right eye hidden variable through calculation according to the t th right eye matching similarity and the (t−1) th right eye attention map by using the ConvLSTM network;

obtain a t th right eye disparity cost according to the t th right eye hidden variable; and

calculate a t th predicted right eye disparity value according to the t th right eye disparity cost, the t th predicted right eye disparity value being used for generating the t th right eye disparity map.

13. The depth information determining apparatus according to claim 12 , wherein

the processing circuitry is configured to calculate the t th left eye hidden variable based on

i′ t =σ( W xi *X′ t +W hi *H′ t−1 +W ci ∘ C′ t−1 ±b i ),

f′ t =σ( W xf *X′ t +W hf *H′ t−1 +W cf ∘ C′ t−1 +b f ),

o′ t =σ( W xo *X′ t +W ho *H′ t−1 +W co ∘ C′ t−1 ±b o ),

C′ t =f′ t ∘ C′ t−1 +t′ t ∘ tanh( W xc *X′ t +W hc *H′ t−1 ±b c ), and

H′ t =o′ t ∘ tanh( C′ t );

i′ t represents a network input gate of a t th left eye recursion, * represents multiplication of vectors, ∘ represents a convolution operation, σ represents a sigmoid function, W xi , W hi , W ci , and b i represent model parameters of the network input gate, X′ t represents the t th left eye matching similarity and the (t−1) th left eye attention map, f′ t represents a forget gate of the t th left eye recursion, W xf , W hf , W cf , and b f represent model parameters of the forget gate, o′ t represents an output gate of the t th left eye recursion, W xo , W ho , W co , and b o represent model parameters of the output gate, C′ t represents a memory cell of the t th left eye recursion, C′ t−1 represents a memory cell of a (t−1) th left eye recursion, tan h represents a hyperbolic tangent function, H′ t−1 represents a (t−1) th left eye hidden variable, and H′ t represents the t th left eye hidden variable;

the processing circuitry is configured to calculate the t th right eye hidden variable based on

i″ t =σ( W xi *X″ t +W hi *H′ t−1 +W ci ∘ C″ t−1 +b i ),

f″ t =σ( W xf *X″ t +W hf *H″ t−1 +W cf ∘ C″ t−1 +b f ),

o″ t =σ( W xo *X″ t +W ho *H″ t−1 +W co ∘ C″ t−1 +b o ),

C″ t =f t ∘ C″ t−1 +i′ t ∘ tanh( W xc *X″ t +W hc *H″ t−1 +b c ), and

H″ t =o″ t ∘ tanh( C″ t ), where

i″ t represents a network input gate of a t th right eye recursion, X″ t represents the t th right eye matching similarity and the (t−1) th right eye attention map, f″ t represents a forget gate of the t th right eye recursion, o″ t represents an output gate of the t th right eye recursion, C″ t represents a memory cell of the t th right eye recursion, C″ t−1 represents a memory cell of a (t−1) th right eye recursion, H″ t−1 represents a (t−1) th right eye hidden variable, and H″ t represents the t th right eye hidden variable.

14. The depth information determining apparatus according to claim 12 , wherein the processing circuitry is configured to

process the t th left eye hidden variable with at least two fully connected layers, to obtain the t th left eye disparity cost; and

process the t th right eye hidden variable with the at least two fully connected layers, to obtain the t th right eye disparity cost.

15. The depth information determining apparatus according to claim 12 , wherein

the processing circuitry is configured to calculate the t th predicted left eye disparity value based on d′*=Σ d=1 Dmax d′*σ(−c′ d );

d′* represents the t th predicted left eye disparity value, D max represents a maximum quantity in different disparity maps, d′ represents a t th left eye disparity value, σ represents a sigmoid function, and c′ d represents the t th left eye disparity cost;

the processing circuitry is configured to calculate the t th predicted right eye disparity value based on d″*=Σ d=1 Dmax d″*σ(−c″ d ); and

d″* represents the t th predicted right eye disparity value, c″ d represents the t th right eye disparity cost, and d″ represents a t th right eye disparity value.

16. The depth information determining apparatus according to claim 13 , wherein

the processing circuitry is configured to calculate the first depth information based on

Z

′

=

B

⁢

f

d

′

*

;

Z′ represents the first depth information, d′* represents the t th predicted left eye disparity value, B represents a binocular camera spacing, and f represents a focal length;

the processing circuitry is configured to calculate the second depth information based on

Z

″

=

B

⁢

f

d

″

*

;

and

Z″ represents the second depth information, and d″* represents the t th predicted right eye disparity value.

17. A non-transitory computer-readable storage medium storing instructions, which when run on a computer, cause the computer to perform a depth information determining method for dual cameras, the depth information determining method comprising:

obtaining a t th left eye matching similarity from a left eye image captured by a first camera of the dual cameras to a right eye image captured by a second camera of the dual cameras, and a t th right eye matching similarity from the right eye image to the left eye image, t being an integer greater than 1;

processing the t th left eye matching similarity and a (t−1) th left eye attention map with a neural network model, to obtain a t th left eye disparity map;

processing the t th right eye matching similarity and a (t−1) th right eye attention map with the neural network model, to obtain a t th right eye disparity map;

determining first depth information according to the t th left eye disparity map; and

determining second depth information according to the t th right eye disparity map.

18. The non-transitory computer-readable storage medium according to claim 17 , the method further comprising:

mapping the t th right eye disparity map to a left eye coordinate system, to obtain a t th left eye mapping disparity map;

generating a t th left eye attention map according to the t th left eye mapping disparity map and the t th left eye disparity map;

mapping the t th left eye disparity map to a right eye coordinate system, to obtain a t th right eye mapping disparity map; and

generating a t th right eye attention map according to the t th right eye mapping disparity map and the t th right eye disparity map.

19. The non-transitory computer-readable storage medium according to claim 18 , wherein after the determining the first depth information and the determining the second depth information, the method further comprises:

obtaining a (t+1) th left eye matching similarity from the left eye image to the right eye image, and a (t+1) th right eye matching similarity from the right eye image to the left eye image;

processing the (t+1) th left eye matching similarity and the t th left eye attention map with the neural network model, to obtain a (t+1) th left eye disparity map;

processing the (t+1) th right eye matching similarity and the t th right eye attention map with the neural network model, to obtain a (t+1) th right eye disparity map;

determining third depth information according to the (t+1) th left eye disparity map; and

determining fourth depth information according to the (t+1) th , right eye disparity map.

20. The non-transitory computer-readable storage medium according to claim 17 , wherein

the processing the t th left eye matching similarity and the (t−1) th left eye attention map includes:

obtaining a t th left eye hidden variable through calculation according to the t th left eye matching similarity and the (t−1) th left eye attention map by using a convolutional long short-term memory (ConvLSTM) network;

obtaining a t th left eye disparity cost according to the t th left eye hidden variable; and

calculating a t th predicted left eye disparity value according to the t th left eye disparity cost, the t th predicted left eye disparity value being used for generating the t th left eye disparity map; and

the processing the t th right eye matching similarity and the (t−1) th right eye attention map includes:

obtaining a t th right eye hidden variable through calculation according to the t th right eye matching similarity and the (t−1) th right eye attention map by using the ConvLSTM network;

obtaining a t th right eye disparity cost according to the t th right eye hidden variable; and

calculating a t th predicted right eye disparity value according to the t th right eye disparity cost, the t th predicted right eye disparity value being used for generating the t th right eye disparity map.