Video depth estimation based on temporal attention

1. A method of depth detection based on a plurality of video frames, the method comprising:

receiving a plurality of input frames comprising a first input frame, a second input frame, and a third input frame respectively corresponding to different capture times;

convolving the first to third input frames to generate a first feature map, a second feature map, and a third feature map corresponding to the different capture times;

calculating a temporal attention map based on the first to third feature maps, the temporal attention map comprising a plurality of weights corresponding to different pairs of feature maps from among the first to third feature maps, each weight of the plurality of weights indicating a similarity level of a corresponding pair of feature maps; and

applying the temporal attention map to the first to third feature maps to generate a feature map with temporal attention.

2. The method of claim 1 , wherein the plurality of weights are based on a learnable value.

3. The method of claim 1 , wherein each weight A ij of the plurality of weights of the temporal attention map is expressed as:

A

ij

=

s

M

r

j

·

M

r

i

∑

i

=

1

3

⁢

C

⁢

s

M

r

j

·

M

r

i

where i and j are index values greater than zero, s is a learnable scaling factor, M r is a reshaped combined feature map based on the first to third feature map, and c represents a number of channels in each of the first to third feature maps.

4. The method of claim 3 , wherein the applying the attention map comprises calculating elements Y i of the feature map with temporal attention as:

Y

i

=

∑

j

=

1

3

⁢

C

⁢

(

A

ij

⁢

M

r

j

)

where i is an index value greater than 0.

5. The method of claim 1 , wherein the input frames are video frames of an input video sequence.

6. The method of claim 1 , wherein the input frames are motion-compensated warped frames based on video frames.

7. The method of claim 1 , further comprising:

receiving a plurality of warped frames comprising a first warped frame, a second warped frame, and a third warped frame; and

spatially dividing each of the first to third warped frames into a plurality of patches,

wherein the first input frame is a patch of the plurality of patches of the first warped frame,

wherein the second input frame is a patch of the plurality of patches of the second warped frame, and

wherein the third input frame is a patch of the plurality of patches of the third warped frame.

8. The method of claim 1 , further comprising:

receiving a first video frame, a second video frame, and a third video frame, the first to third video frames being successive frames of a video sequence;

compensating for motions between the first to third video frames based on optical flow to generate the first to third input frames; and

generating a depth map based on the feature map with temporal attention, the depth map comprising depth values of pixels of the second video frame.

9. The method of claim 8 , wherein the compensating for motions comprises:

determining optical flow of pixels of the second video frame based on pixels of the first and third video frames; and

image warping the first to third input frames based on the determined optical flow.

10. The method of claim 1 , further comprising:

receiving a first video frame, a second video frame, and a third video frame, the first to third video frames being successive frames of a video sequence;

generating a first depth map, a second depth map, and a third depth map based on the first to third video frames;

compensating for motions between the first to third depth maps based on optical flow to generate the first to third input frames; and

convolving the feature map with temporal attention to generate a depth map, the depth map comprising depth values of pixels of the second video frame.

11. The method of claim 10 , wherein the first to third input frames are warped depth maps corresponding to the first to third depth maps.

12. The method of claim 10 , wherein generating the first to third depth maps comprises:

generating the first depth map based on the first video frame;

generating the second depth map based on the second video frame; and

generating the third depth map based on the third video frame.

13. A method of depth detection based on a plurality of video frames, the method comprising:

identifying a plurality of warped frames comprising a first warped frame, a second warped frame, and a third warped frame corresponding to different capture times;

dividing each of the first to third warped frames into a plurality of patches comprising a first patch;

convolving the first patch of the first warped frame, the first patch of the second warped frame, and the first patch of the third warped frame to generate a first feature map, a second feature map, and a third feature map corresponding to the different capture times;

calculating a temporal attention map based on the first to third feature maps, the temporal attention map comprising a plurality of weights corresponding to different pairs of feature maps from among the first to third feature maps, each weight of the plurality of weights indicating a similarity level of a corresponding pair of feature maps; and

applying the temporal attention map to the first to third feature maps to generate a feature map with temporal attention.

14. The method of claim 13 , wherein the plurality of warped frames are motion compensated video frames.

15. The method of claim 13 , wherein the plurality of warped frames are motion compensated depth maps corresponding to a plurality of input video frames of a video sequence.

16. The method of claim 13 , further comprising:

receiving a first video frame, a second video frame, and a third video frame, the first to third video frames being successive frames of a video sequence;

compensating for motions between the first to third video frames based on optical flow to generate the first to third warped frames; and

generating a depth map based on the feature map with temporal attention, the depth map comprising depth values of pixels of the second video frame.

17. The method of claim 16 , wherein the compensating for motions comprises:

determining optical flow of pixels of the second video frame based on pixels of the first and third video frames; and

image warping the first to third video frames based on the determined optical flow.

18. The method of claim 13 , further comprising:

receiving a first video frame, a second video frame, and a third video frame, the first to third video frames being successive frames of a video sequence;

generating a first depth map, a second depth map, and a third depth map based on the first to third video frames;

compensating for motions between the first to third depth maps based on optical flow to generate the first to third warped maps; and

convolving the feature map with temporal attention to generate a depth map, the depth map comprising depth values of pixels of the second video frame.

19. The method of claim 18 , wherein the first to third warped maps are warped depth maps corresponding to the first to third depth maps.

20. A system for depth detection based on a plurality of video frames, the system comprising:

a processor; and

a processor memory local to the processor, wherein the processor memory has stored thereon instructions that, when executed by the processor, cause the processor to perform:

receiving a plurality of input frames comprising a first input frame, a second input frame, and a third input frame respectively corresponding to different capture times;

convolving the first to third input frames to generate a first feature map, a second feature map, and a third feature map corresponding to the different capture times;

calculating a temporal attention map based on the first to third feature maps, the temporal attention map comprising a plurality of weights corresponding to different pairs of feature maps from among the first to third feature maps, each weight of the plurality of weights indicating a similarity level of a corresponding pair of feature maps; and

applying the temporal attention map to the first to third feature maps to generate a feature map with temporal attention.