Video processing method, device, equipment and storage medium

1 . A video processing method, comprising:

dividing an original video sequence into a plurality of image blocks;

inputting the plurality of image blocks into an encoding network model to obtain implicit representation video parameters corresponding to the original video sequence, wherein the implicit representation video parameters are abstract representations of the original video sequence, the implicit representation video parameters are compressed representations of the original video sequence;

inputting the implicit representation video parameters into a reconstruction network model to obtain a reconstructed video sequence corresponding to the original video sequence;

calculating a loss function value based on the original video sequence and the reconstructed video sequence; and

based on the loss function value, the network parameters of the encoding network model be adjusted to adjust the implicit representation video parameters until the implicit representation video parameters meet preset requirements, and obtaining target implicit representation video parameters;

wherein before inputting the plurality of image blocks into the encoding network model to obtain the implicit representation video parameters corresponding to the original video sequence, the video processing method further comprises:

determining a model structure of the encoding network model based on sequence characteristics corresponding to the original video sequence

or a first performance corresponding to the encoding network model

or the original video sequence and a first performance corresponding to the encoding network model;

wherein the step of determining the model structure of the encoding network model based on the sequence characteristics corresponding to the original video sequence or the first performance of the encoding network model or the original video sequence and a first performance corresponding to the encoding network model comprises:

determining a number of first intermediate layers of the encoding network model and a measured bitrate of the encoding network model in the first intermediate layers;

when the measured bitrate is less than a minimum threshold of a preset bitrate range, increasing the number of first intermediate layers until the measured bitrate falls within the preset bitrate range; and

when the measured bitrate is greater than a maximum threshold of the preset bitrate range, reducing the number of first intermediate layers until the measured bitrate falls within the preset bitrate range;

wherein the video processing method adjusts the implicit representation video parameters based on the loss function value, continues training until preset training completion conditions are met, and obtains the target implicit representation video parameters, comprising:

in each iteration process, calculating a rate distortion corresponding to the implicit representation video parameters; and

when the rate distortion is less than a rate distortion threshold, continuing training until the rate distortion corresponding to the implicit representation video parameters is greater than or equal to the rate distortion threshold or training times reach a training times threshold to obtain the target implicit representation video parameters;

the process of encoding multiple image blocks to implicitly represent video parameters be implemented as follows:

extract the image features corresponding to multiple image blocks, fuse multiple image features, and use the encoder to map the fused image features to the low dimensional implicit space representation, that is, encode the video parameters into the implicit representation;

image features include spatial features and temporal features; convolutional neural network be used to extract the spatial features.

2 . The video processing method according to claim 1 , wherein the step of determining the model structure of the encoding network model based on the sequence characteristics corresponding to the original video sequence or the first performance of the encoding network model or the original video sequence and a first performance corresponding to the encoding network model comprises:

determining at least one adjacent frame in the original video sequence with a frame difference value greater than a frame difference threshold; and

adding learning parameters in a network layer corresponding to the at least one adjacent frame.

3 . The video processing method according to claim 1 , wherein before inputting the implicit representation video parameters into the reconstruction network model to obtain the reconstructed video sequence corresponding to the original video sequence, the video processing method further comprises:

determining a model structure of the reconstruction network model based on video information corresponding to each of the plurality of image blocks

or based on a second performance corresponding to the reconstruction network model

or based on video information corresponding to each of the plurality of image blocks and a second performance corresponding to the reconstruction network model.

4 . The video processing method according to claim 1 , wherein a model structure of the reconstruction network model is determined based on video information corresponding to each of the plurality of image blocks and/or a second performance corresponding to the reconstruction network model by:

adding a second middle layer group of the reconstructed network model, wherein the second middle layer group is used to execute an iterative process, and an input of the second middle layer group is consistent;

or deepening at least one third intermediate layer of the reconstructed network model;

or widening at least one fourth intermediate layer of the reconstructed network model;

or adding a second middle layer group of the reconstructed network model, wherein the second middle layer group is used to execute an iterative process, and an input of the second middle layer group is consistent; and deepening at least one third intermediate layer of the reconstructed network model;

or adding a second middle layer group of the reconstructed network model, wherein the second middle layer group is used to execute an iterative process, and an input of the second middle layer group is consistent; and widening at least one fourth intermediate layer of the reconstructed network model;

or deepening at least one third intermediate layer of the reconstructed network model; and widening at least one fourth intermediate layer of the reconstructed network model;

or adding a second middle layer group of the reconstructed network model, wherein the second middle layer group is used to execute an iterative process, and an input of the second middle layer group is consistent; and deepening at least one third intermediate layer of the reconstructed network model; and widening at least one fourth intermediate layer of the reconstructed network model; and

calculating first order edge information and second order curvature information corresponding to the plurality of image blocks, and adding the first order edge information and the second order curvature information corresponding to the plurality of image blocks to a network layer corresponding to the plurality of image blocks.

5 . A video processing device, comprising:

a partitioning module, configured for partitioning an original video sequence to obtain a plurality of image blocks;

an input module, configured for:

inputting the plurality of image blocks into an encoding network model to obtain implicit representation video parameters corresponding to the original video sequence, wherein the implicit representation video parameters are abstract representations of the original video sequence, the implicit representation video parameters are compressed representations of the original video sequence; and

inputting the implicit representation video parameters into a reconstruction network model to obtain a reconstructed video sequence corresponding to the original video sequence;

a calculation module, configured for calculating a loss function value based on the original video sequence and the reconstructed video sequence; and

an adjustment module, configured for adjusting the network parameters of the encoding network model to adjusted the implicit representation video parameters based on the loss function value until the implicit representation video parameters meet preset requirements to obtain target implicit representation video parameters;

wherein the video processing method adjusts the implicit representation video parameters based on the loss function value, continues training until preset training completion conditions are met, and obtains the target implicit representation video parameters, comprising:

in each iteration process, calculating a rate distortion corresponding to the implicit representation video parameters; and

when the rate distortion is less than a rate distortion threshold, continuing training until the rate distortion corresponding to the implicit representation video parameters is greater than or equal to the rate distortion threshold or training times reach a training times threshold to obtain the target implicit representation video parameters;

the process of encoding multiple image blocks to implicitly represent video parameters be implemented as follows:

extract the image features corresponding to multiple image blocks, fuse multiple image features, and use the encoder to map the fused image features to the low dimensional implicit space representation, that is, encode the video parameters into the implicit representation;

image features include spatial features and temporal features; convolutional neural network be used to extract the spatial features.

6 . An electronic device, comprising a memory, bus and communication interface, a processor and a computer program stored on the memory and configured to run on the processor, wherein the processor runs the computer program to realize the video processing method according to claim 1 ; processor, communication interface and memory are connected through bus, the memory include non volatile memory.

7 . A computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, the computer-readable storage medium include electrically erasable programmable read-only memory; and the computer program is executed by a processor to realize the video processing method according to claim 1 .

8 . The electronic device according to claim 6 , wherein the step of determining the model structure of the encoding network model based on the sequence characteristics corresponding to the original video sequence and/or the first performance of the encoding network model comprises:

determining at least one adjacent frame in the original video sequence with a frame difference value greater than a frame difference threshold; and

adding learning parameters in a network layer corresponding to the at least one adjacent frame.

9 . The electronic device according to claim 6 , wherein before inputting the implicit representation video parameters into the reconstruction network model to obtain the reconstructed video sequence corresponding to the original video sequence, the video processing method further comprises:

determining a model structure of the reconstruction network model based on video information corresponding to each of the plurality of image blocks

or based on a second performance corresponding to the reconstruction network model

or based on video information corresponding to each of the plurality of image blocks and a second performance corresponding to the reconstruction network model.

10 . The electronic device according to claim 6 , wherein a model structure of the reconstruction network model is determined based on video information corresponding to each of the plurality of image blocks and/or a second performance corresponding to the reconstruction network model by:

adding a second middle layer group of the reconstructed network model, wherein the second middle layer group is used to execute an iterative process, and an input of the second middle layer group is consistent;

or deepening at least one third intermediate layer of the reconstructed network model;

or widening at least one fourth intermediate layer of the reconstructed network model;

or adding a second middle layer group of the reconstructed network model, wherein the second middle layer group is used to execute an iterative process, and an input of the second middle layer group is consistent; and deepening at least one third intermediate layer of the reconstructed network model;

or adding a second middle layer group of the reconstructed network model, wherein the second middle layer group is used to execute an iterative process, and an input of the second middle layer group is consistent; and widening at least one fourth intermediate layer of the reconstructed network model;

or deepening at least one third intermediate layer of the reconstructed network model; and widening at least one fourth intermediate layer of the reconstructed network model;

or adding a second middle layer group of the reconstructed network model, wherein the second middle layer group is used to execute an iterative process, and an input of the second middle layer group is consistent; and deepening at least one third intermediate layer of the reconstructed network model; and widening at least one fourth intermediate layer of the reconstructed network model; and

calculating first order edge information and second order curvature information corresponding to the plurality of image blocks, and adding the first order edge information and the second order curvature information corresponding to the plurality of image blocks to a network layer corresponding to the plurality of image blocks.

11 . The computer-readable storage medium according to claim 7 , wherein the step of determining the model structure of the encoding network model based on the sequence characteristics corresponding to the original video sequence and/or the first performance of the encoding network model comprises:

determining at least one adjacent frame in the original video sequence with a frame difference value greater than a frame difference threshold; and

adding learning parameters in a network layer corresponding to the at least one adjacent frame.

12 . The computer-readable storage medium according to claim 7 , wherein before inputting the implicit representation video parameters into the reconstruction network model to obtain the reconstructed video sequence corresponding to the original video sequence, the video processing method further comprises:

determining a model structure of the reconstruction network model based on video information corresponding to each of the plurality of image blocks

or based on a second performance corresponding to the reconstruction network model

or based on video information corresponding to each of the plurality of image blocks and a second performance corresponding to the reconstruction network model.