Video processing method and device, electronic apparatus, and readable storage medium
View Patent ↗The present disclosure provides a video processing method, a video processing device, an electronic apparatus, and a readable storage medium. The video processing method includes: obtaining input data; and inputting the input data into a video processing model to obtain output video data. A resolution and/or a duration of the output video data is not equal to a resolution and/or a duration of the input data, the video processing model includes a plurality of generators arranged in sequence and corresponding to different image resolutions, each generator includes a transposed 3D convolution unit and a plurality of first 3D convolutional layers, the transposed 3D convolution unit is configured to generate first output data in accordance with the input data and intermediate processing data of the generator.
1 . A video processing method, comprising:
obtaining input data, the input data comprising picture data and/or video data; and
inputting the input data into a video processing model to obtain output video data;
wherein a resolution and/or a duration of the output video data is not equal to a resolution and/or a duration of the input data, the video processing model comprises a generative adversarial network, the generative adversarial network comprises a generator and a discriminator, the generator comprises a plurality of generator levels arranged in sequence and corresponding to different image resolutions, each generator level in the plurality of generator levels comprises a transposed three-dimensional (3D) convolution unit and a plurality of first 3D convolutional layers, the transposed 3D convolution unit of a current generator level of the plurality of generator levels is configured to generate first output data in accordance with the input data and intermediate processing data of the current generator level, the output video data is obtained in accordance with the first output data, and the intermediate processing data is obtained through inputting the input data into the plurality of first 3D convolutional layers of the current generator level;
wherein each generator level in the plurality of generator levels further comprises a second 3D convolutional layer, the second 3D convolutional layer is configured to adjust the first output data to obtain second output data, and a step size of at least one dimension of the second 3D convolutional layer is greater than a step size of the transposed 3D convolution unit; and
wherein the second 3D convolutional layer comprises a first step size corresponding to a time dimension and a second step size corresponding to an output size dimension, and the transposed 3D convolution unit comprises a third step size corresponding to the time dimension and a fourth step size corresponding to the output size dimension.
2 . The video processing method according to claim 1 , wherein a ratio of the first step size to the third step size is not equal to a ratio of the second step size to the fourth step size.
3 . The video processing method according to claim 1 , wherein the quantity of the first 3D convolutional layers is four.
4 . The video processing method according to claim 3 , wherein each of the first 3D convolutional layers comprises a 3D convolution unit and a residual layer arranged in sequence.
5 . The video processing method according to claim 1 , wherein in the case that the input data comprises the video data and the duration of the output video data is not equal to the duration of the input data, a plurality of first images corresponds to a second image, wherein the plurality of the first images is a plurality of different images in the input data, and the second image is one image in the output video data comprising features of the plurality of the first images.
6 . The video processing method according to claim 1 , wherein the video processing model is obtained through the generative adversarial network after performing model training;
wherein the generator is a model with a low-resolution image as an input and a high-resolution video sequence as an output;
the discriminator is a model with an image as an input and a discrimination result for the image as an output; and
a loss function for the model training is determined in accordance with an adversarial loss between the generator and the discriminator, and an input and output reconstruction loss of the generator.
7 . The video processing method according to claim 1 , wherein in the case that the input data comprises the video data, prior to inputting the input data into the video processing model to obtain the output video data, the video processing method further comprises:
dividing the input data into a plurality of video sequences at a predetermined time interval;
extracting 3D-patches of the video sequence, each of pixels in the video sequence being located in at least one of the 3D-patches, and at least a part of the pixels being located in a plurality of the 3D-patches; and
taking the 3D-patches as an input of the video processing model.
8 . A non-transient readable storage medium, configured to store a program, wherein the program is executed by a processor so as to implement the steps in the image processing method according to claim 1 .
9 . An electronic apparatus, comprising a memory, a processor, and a program stored in the memory and executed by the processor, wherein the program is executed by the processor so as to:
obtain input data, the input data comprising picture data and/or video data; and
input the input data into a video processing model to obtain output video data;
wherein a resolution and/or a duration of the output video data is not equal to a resolution and/or a duration of the input data, the video processing model comprises a generative adversarial network, the generative adversarial network comprises a generator and a discriminator, the generator comprises a plurality of generator levels arranged in sequence and corresponding to different image resolutions, each generator level in the plurality of generator levels comprises a transposed 3D convolution unit and a plurality of first 3D convolutional layers, the transposed 3D convolution unit of a current generator level of the plurality of generator levels is configured to generate first output data in accordance with the input data and intermediate processing data of the current generator level, the output video data is obtained in accordance with the first output data, and the intermediate processing data is obtained through inputting the input data into the plurality of first 3D convolutional layers of the current generator level;
wherein each generator level in the plurality of generator levels further comprises a second 3D convolutional layer, the second 3D convolutional layer is configured to adjust the first output data to obtain second output data, and a step size of at least one dimension of the second 3D convolutional layer is greater than a step size of the transposed 3D convolution unit;
wherein the second 3D convolutional layer comprises a first step size corresponding to a time dimension and a second step size corresponding to an output size dimension, and the transposed 3D convolution unit comprises a third step size corresponding to the time dimension and a fourth step size corresponding to the output size dimension.
10 . The electronic apparatus according to claim 9 , wherein a ratio of the first step size to the third step size is not equal to a ratio of the second step size to the fourth step size.
11 . The electronic apparatus according to claim 9 , wherein the quantity of the first 3D convolutional layers is four.
12 . The electronic apparatus according to claim 11 , wherein each of the first 3D convolutional layers comprises a 3D convolution unit and a residual layer arranged in sequence.
13 . The electronic apparatus according to claim 9 , wherein in the case that the input data comprises the video data and the duration of the output video data is not equal to the duration of the input data, a plurality of first images corresponds to a second image, wherein the plurality of the first images is a plurality of different images in the input data, and the second image is one image in the output video data comprising features of the plurality of the first images.
14 . The electronic apparatus according to claim 9 , wherein the video processing model is obtained through the generative adversarial network after performing model training;
wherein the generator is a model with a low-resolution image as an input and a high-resolution video sequence as an output;
the discriminator is a model with an image as an input and a discrimination result for the image as an output; and
a loss function for the model training is determined in accordance with an adversarial loss between the generator and the discriminator, and an input and output reconstruction loss of the generator.
15 . The electronic apparatus according to claim 9 , wherein in the case that the input data comprises the video data, prior to inputting the input data into the video processing model to obtain the output video data, the processor is further configured to:
divide the input data into a plurality of video sequences at a predetermined time interval;
extract 3D-patches of the video sequence, each of pixels in the video sequence being located in at least one of the 3D-patches, and at least a part of the pixels being located in a plurality of the 3D-patches; and
take the 3D-patches as an input of the video processing model.