Video processing method and video processing apparatus

1 . A video processing method comprising:

obtaining a clip representation based on a frame index and a global representation of an input video;

obtaining a residual frame representation based on the frame index and a residual representation of the input video using a residual neural network; and

outputting a final frame representation generated based on the clip representation and the residual frame representation.

2 . The video processing method of claim 1 , wherein the obtaining of the clip representation comprises:

dividing the global representation of the input video into a plurality of clips; and

selecting one or more first clips from the plurality of clips based on the frame index.

3 . The video processing method of claim 2 , wherein the obtaining of the clip representation further comprises:

setting one or more second clips, which are not selected as the one or more first clips from the plurality of clips, to zero tensor; and

combining the one or more first clips and the one or more second clips.

4 . The video processing method of claim 2 , wherein the dividing comprises:

determining whether the input video is static or dynamic; and

based on the determination, adaptively adjusting a number of the plurality of clips.

5 . The video processing method of claim 4 , wherein the determining whether the input video is static or dynamic comprises:

calculating a Mean Squared Error (MSE) between a first frame and a second frame of the input video, and

based on the calculated MSE being less than or equal to a threshold value, determining that the input video is static, and based on the calculated MSE exceeding the threshold value, determining that the input video is dynamic.

6 . The video processing method of claim 4 , wherein the number of the plurality of clips is a first number in a case in which the input video is dynamic, and the number of the plurality of clips is a second number in a case in which the input video is static, the first number being larger than the second number.

7 . The video processing method of claim 2 , wherein the selecting of the one or more first clips comprises selecting the one or more first clips based on the frame index and a distance between the plurality of clips.

8 . The video processing method of claim 7 , wherein the selecting of the one or more first clips comprises selecting a first clip and a second clip, among the plurality of clips, that are closer to the frame index than other clips in the plurality of clips, and

wherein the first clip is on a first side of the frame index and the second clip is on a second side of the frame index opposite to the first side.

9 . The video processing method of claim 2 , wherein the dividing comprises:

detecting a scene change point in the input video; and

dividing the plurality of clips with different depths based on the detected scene change point.

10 . The video processing method of claim 1 , wherein the outputting of the residual frame representation comprises performing channel attention by using the residual representation as an input.

11 . The video processing method of claim 10 , wherein the outputting of the residual frame representation further comprises performing normalization by using the frame index and a result of the channel attention as input.

12 . The video processing method of claim 11 , wherein the normalization comprises Instance Normalization.

13 . The video processing method of claim 11 , wherein the outputting of the residual frame representation further comprises performing at least one of convolution, pixel shuffle, or activation based on a result of the normalization.

14 . The video processing method of claim 1 , further comprising outputting a predicted frame by inputting the final frame representation to a decoder.

15 . The video processing method of claim 1 , further comprising training a clip selection model and the residual neural network by using the predicted frame and a ground truth frame.

16 . A video processing apparatus comprising:

a memory storing one or more instructions; and

a processor configured to execute the one or more instructions to:

obtain a clip representation based on a frame index and a global representation of an input video;

obtain a residual frame representation based on the frame index and a residual representation of the input video using a residual neural network; and

output a final frame representation generate based on the clip representation and the residual frame representation.

17 . The video processing apparatus of claim 16 , wherein the processor is further configured to generate the clip representation by:

adaptively dividing the global representation into a plurality of clips based on at least one of whether the input video is static or dynamic or whether there is a scene change point, and

selecting one or more of the plurality of clips.

18 . The video processing apparatus of claim 16 , further comprising a decoder configured to output a predicted frame based on the final frame representation.

19 . The video processing apparatus of claim 16 , wherein the processor is further configured to train a clip selection model and the residual neural network based on the predicted frame and a ground truth frame.

20 . An electronic device comprising:

a storage configured to store a video;

a video processing apparatus configured to process the video and output a result of the processing; and

wherein the video processing apparatus comprises:

a memory storing one or more instructions; and

a processor configured to execute the one or more instructions to:

obtain a clip representation based on a frame index and a global representation of an input video;

obtain a residual frame representation based on the frame index and a residual representation of the input video using a residual neural network; and

output a final frame representation generate based on the clip representation and the residual frame representation.