Granted Patent B2

US 12,200,248 · App. 18/010,356 · Granted Jan 14, 2025

Inventors: Wei Yang (Guangdong, CN); Ke Xu (Guangdong, CN); Dehui Kong (Guangdong, CN); Jianjun Song (Guangdong, CN); Fang Zhu (Guangdong, CN)

Assignee: SANECHIPS TECHNOLOGY CO., LTD.

H04N19/59G06T5/50G06T7/11H04N19/51G06T2207/20221

Abstract

Claims (56)

1. A video processing method, comprising:

coding and decoding an original video to obtain a mixed resolution video, wherein the mixed resolution video comprises a first resolution frame and a second resolution frame each corresponding to a key frame in the original video, and a third resolution frame corresponding to a non-key frame in the original video, wherein the first resolution frame has a resolution higher than a resolution of the second resolution frame or a resolution of the third resolution frame; and

amplifying, according to the first resolution frame and the second resolution frame corresponding to the key frame, the third resolution frame corresponding to the non-key frame in the mixed resolution video to output an amplified video, wherein the amplified video comprises the first resolution frame corresponding to the key frame, and an amplified target frame corresponding to the non-key frame,

wherein amplifying, according to the first resolution frame and the second resolution frame corresponding to the key frame, the third resolution frame in the mixed resolution video comprises:

inputting, for each non-key frame, the third resolution frame of the non-key frame, the third resolution frame before amplification and a high resolution frame after amplification of a previous non-key frame of the non-key frame, and first resolution frames and second resolution frames corresponding to the previous key frame and a next key frame adjacent to the non-key frame into a video amplification network;

amplifying, according to the third resolution frame before amplification and the high resolution frame after amplification of the previous non-key frame of the non-key frame, and the first resolution frames and the second resolution frames corresponding to the previous key frame and the next key frame adjacent to the non-key frame, the third resolution frame of the non-key frame through the video amplification network to obtain the amplified target frame corresponding to the non-key frame.

2. The method according to claim 1 , wherein coding and decoding the original video to obtain the mixed resolution video comprises:

detecting the key frame in the original video;

generating a residual plot and the second resolution frame of the key frame, and performing down-sampling on the non-key frame to obtain the third resolution frame; and

performing video coding and decoding on the residual plot, the second resolution frame and the third resolution frame to obtain the mixed resolution video.

3. The method according to claim 2 , wherein generating the residual plot and the second resolution frame of the key frame comprises:

performing down-sampling on the key frame to obtain the second resolution frame of the key frame; and

performing up-sampling on the key frame to obtain a temporary frame of the first resolution frame of the key frame, and subtracting the temporary frame of the first resolution frame from the key frame to obtain the residual plot of the key frame.

4. The method according to claim 2 , wherein performing video coding and decoding on the residual plot, the second resolution frame and the third resolution frame to obtain the mixed resolution video comprises:

coding the residual plot, the second resolution frame and the third resolution frame with a coder, to obtain a residual code stream, a code stream of the second resolution frame and a code stream of the third resolution frame;

decoding the residual code stream, the code stream of the second resolution frame and the code stream of the third resolution frame with a decoder, to respectively obtain the residual plot, the second resolution frame and the third resolution frame; and

performing up-sampling on the second resolution frame to obtain an amplified fourth resolution frame, and adding the fourth resolution frame and the residual plot to obtain the first resolution frame of the key frame.

5. The method according to claim 1 , wherein coding and decoding the original video to obtain the mixed resolution video comprises:

detecting the key frame in the original video;

equally dividing the key frame into a plurality of sub-images, wherein each sub-image has a same resolution as a resolution of the second resolution frame, and performing down-sampling on the non-key frame to obtain the third resolution frame; and

inputting the plurality of sub-images and the third resolution frame of the non-key frame into a video codec, wherein the video codec comprises a coder and a decoder;

coding the plurality of sub-images and the third resolution frame of the non-key frame with the coder, to obtain code streams corresponding to the plurality of sub-images and the third resolution frame of the non-key frame;

splicing the plurality of sub-images according to the code streams of the plurality of sub-images with the decoder, to obtain the first resolution frame of the key frame, and decoding the code stream of the third resolution frame to obtain the third resolution frame.

6. The method according to claim 1 , wherein the video amplification network comprises a motion estimation sub-network, a motion compensation sub-network, and an amplification sub-network; and

amplifying, according to the third resolution frame before amplification and the high resolution frame after amplification of the previous non-key frame of the non-key frame, and the first resolution frames and the second resolution frames corresponding to the previous key frame and the next key frame adjacent to the non-key frame, the third resolution frame of the non-key frame through the video amplification network to obtain the amplified target frame corresponding to the non-key frame comprises:

respectively performing, based on the third resolution frame of the non-key frame, motion estimation on the third resolution frame before amplification of the previous non-key frame of the non-key frame, and the second resolution frames corresponding to the previous key frame and the next key frame adjacent to the non-key frame through the motion estimation sub-network;

respectively performing, according to the high resolution frame after amplification of the previous non-key frame and the first resolution frames corresponding to the previous key frame and the next key frame adjacent to the non-key frame, motion compensation on each frame subjected to motion estimation and an up-sampled motion vector to obtain an intermediate high resolution feature map, and converting the intermediate high resolution feature map into a low resolution feature map through shuffle; and

amplifying, according to the low resolution feature map, the third resolution frame through the amplification sub-network to obtain the amplified target frame corresponding to the non-key frame.

7. The method according to claim 6 , wherein respectively performing, according to the high resolution frame after amplification of the previous non-key frame and the first resolution frames corresponding to the previous key frame and the next key frame adjacent to the non-key frame, motion compensation on each frame subjected to motion estimation and the up-sampled motion vector to obtain the intermediate high resolution feature map comprises:

respectively performing up-sampling on motion vectors MV t-1 LR , MV k1 LR and MV k2 LR corresponding to frames subjected to motion estimation to obtain MV t-1 SR , MV k1 SR and MV k2 SR ; wherein MV t-1 LR , MV k1 LR and MV k2 LR are respectively amplified motion vectors of the third resolution frame before amplification of the previous non-key frame, and the second resolution frames corresponding to the previous key frame and the next key frame adjacent to the non-key frame relative to the third resolution frame of the non-key frame obtained by motion estimation;

warping, based on MV t-1 SR , MV k1 SR and MV k2 SR , the high resolution frame after amplification of the previous non-key frame and the first resolution frames corresponding to the previous key frame and the next key frame adjacent to the non-key frame through a warp operation, to obtain the warped frames I t-1→t SR , I k1→t SR and I k2→t SR , respectively;

inputting into the I t-1→t SR , I k1→t SR , I k2→t SR , MV t-1 SR , MV k1 SR and MV k2 SR into the compensation sub-network, and obtaining the intermediate high resolution feature map after motion compensation through the compensation sub-network.

8. The method according to claim 1 , wherein in a case where the non-key frame is a first non-key frame after the adjacent previous key frame, the second resolution frame of the previous key frame is used as the third resolution frame before amplification of the previous non-key frame of the non-key frame, and the first resolution frame of the previous key frame is used as the high resolution frame after amplification of the previous non-key frame of the non-key frame.

9. The method according to claim 7 , wherein before amplifying, according to the first resolution frame and the second resolution frame corresponding to the key frame, the third resolution frame in the mixed resolution video, the method further comprises:

training, based on sample videos, the video amplification network according to a first loss function and a second loss function, wherein the video amplification network is configured to amplifying, according to the first resolution frame and the second resolution frame corresponding to the key frame, the third resolution frame in the mixed resolution video;

wherein the first loss function is determined from a mean square error of an original high resolution frame of a non-key frame between two adjacent key frames in the original video, and a frame obtained by performing down-sampling on the original high resolution frame and then amplifying through the video amplification network;

the second loss function is determined from a mean square error of the original high resolution frame of the non-key frame, the frame I t-1→t SR obtained from the third resolution frame of the non-key frame and the third resolution frame of the previous non-key frame after motion estimation, motion vector up-sampling, and warping, the frame I k1→t SR obtained from the third resolution frame of the non-key frame and the second resolution frame of the adjacent previous key frame after motion estimation, motion vector up-sampling, and warping, and the frame I k2→t SR obtained from the third resolution frame of the non-key frame and the second resolution frame of the adjacent next key frame after motion estimation, motion vector up-sampling, and warping.

10. The method according to claim 9 , wherein the video amplification network comprises the motion estimation sub-network, the motion compensation sub-network, and the amplification sub-network; and

training the video amplification network according to the first loss function and the second loss function comprises:

training the video amplification network according to the first loss function, and training the motion estimation sub-network according to the second loss function, until the first loss function and the second loss function meet set conditions.

11. The method according to claim 9 , wherein before training the video amplification network according to the first loss function and the second loss function, the method further comprises:

pre-processing the sample videos to obtain a sample video with a preset resolution.

12. A device, comprising:

one or more processors;

a storage having one or more programs stored thereon which, when executed by the one or more processors, cause the one or more processors to implement a video processing method, comprising:

coding and decoding an original video to obtain a mixed resolution video, wherein the mixed resolution video comprises a first resolution frame and a second resolution frame each corresponding to a key frame in the original video, and a third resolution frame corresponding to a non-key frame in the original video, wherein the first resolution frame has a resolution higher than a resolution of the second resolution frame or a resolution of the third resolution frame; and

amplifying, according to the first resolution frame and the second resolution frame corresponding to the key frame, the third resolution frame corresponding to the non-key frame in the mixed resolution video to output an amplified video, wherein the amplified video comprises the first resolution frame corresponding to the key frame, and an amplified target frame corresponding to the non-key frame,

wherein amplifying, according to the first resolution frame and the second resolution frame corresponding to the key frame, the third resolution frame in the mixed resolution video comprises:

inputting, for each non-key frame, the third resolution frame of the non-key frame, the third resolution frame before amplification and a high resolution frame after amplification of a previous non-key frame of the non-key frame, and first resolution frames and second resolution frames corresponding to the previous key frame and a next key frame adjacent to the non-key frame into a video amplification network;

amplifying, according to the third resolution frame before amplification and the high resolution frame after amplification of the previous non-key frame of the non-key frame, and the first resolution frames and the second resolution frames corresponding to the previous key frame and the next key frame adjacent to the non-key frame, the third resolution frame of the non-key frame through the video amplification network to obtain the amplified target frame corresponding to the non-key frame.

13. A non-transitory computer-readable storage medium having a computer program stored thereon which, when executed by a processor, causes the processor to implement a video processing method, comprising:

coding and decoding an original video to obtain a mixed resolution video, wherein the mixed resolution video comprises a first resolution frame and a second resolution frame each corresponding to a key frame in the original video, and a third resolution frame corresponding to a non-key frame in the original video, wherein the first resolution frame has a resolution higher than a resolution of the second resolution frame or a resolution of the third resolution frame; and

amplifying, according to the first resolution frame and the second resolution frame corresponding to the key frame, the third resolution frame corresponding to the non-key frame in the mixed resolution video to output an amplified video, wherein the amplified video comprises the first resolution frame corresponding to the key frame, and an amplified target frame corresponding to the non-key frame,

wherein amplifying, according to the first resolution frame and the second resolution frame corresponding to the key frame, the third resolution frame in the mixed resolution video comprises:

inputting, for each non-key frame, the third resolution frame of the non-key frame, the third resolution frame before amplification and a high resolution frame after amplification of a previous non-key frame of the non-key frame, and first resolution frames and second resolution frames corresponding to the previous key frame and a next key frame adjacent to the non-key frame into a video amplification network;

amplifying, according to the third resolution frame before amplification and the high resolution frame after amplification of the previous non-key frame of the non-key frame, and the first resolution frames and the second resolution frames corresponding to the previous key frame and the next key frame adjacent to the non-key frame, the third resolution frame of the non-key frame through the video amplification network to obtain the amplified target frame corresponding to the non-key frame.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (15)