Method and apparatus for temporal resampling
Methods and apparatuses are provided for encoding and decoding video data based on a supplemental enhancement information (SEI) message. An exemplary method includes: generating a reconstructed frame sequence based on a compressed video; decoding a supplemental enhancement information (SEI) message with respect to the reconstructed frame sequence, according to the compressed video; and performing temporal upsampling to the reconstructed frame sequence based on the SEI message by using a neural network.
1 . A method for decoding video data, comprising:
generating a reconstructed frame sequence based on a compressed video;
decoding a supplemental enhancement information (SEI) message with respect to the reconstructed frame sequence, according to the compressed video; and
performing temporal upsampling to the reconstructed frame sequence based on the SEI message by using a neural network;
wherein the SEI message comprises a first indicator for indicating an upsampling number, and a second indicator for indicating an upsampling strength describing a relationship between the upsampling number and a referencing range, and
wherein the referencing range is determined based on the first indicator and the second indicator.
2 . The method according to claim 1 , wherein performing temporal upsampling to the reconstructed frame sequence comprises:
determining, based on the SEI message, the upsampling number and the referencing range relative to a first frame and a second frame within the reconstructed frame sequence, the second frame being adjacent to the first frame; and
interpolating, via the neural network, the upsampling number of frames between the first frame and the second frame by referencing within the referencing range.
3 . The method according to claim 1 , wherein the referencing range is a product of the upsampling number and the upsampling strength when the second indicator is not equal to zero.
4 . The method according to claim 3 , wherein the SEI message further comprises a third indicator for indicating the referencing range, and the referencing range is determined based on the third indicator when the second indicator is equal to zero.
5 . The method according to claim 1 , wherein the referencing range is a product of the upsampling strength plus one and the upsampling number.
6 . The method according to claim 2 , wherein the SEI message comprises a third indicator for indicating the referencing range.
7 . The method according to claim 6 , wherein a ratio of the referencing range to the upsampling number is within a predetermined range.
8 . The method according to claim 7 , wherein the predetermined range is [ 1/16, 16].
9 . The method according to claim 2 , wherein the SEI message comprises a fourth indicator for indicating whether to perform temporal resampling to the reconstructed frame sequence.
10 . The method according to claim 2 , wherein interpolating the upsampling number of frames between the first frame and the second frame by referencing within the referencing range comprises:
generating, based on the upsampling number and the referencing range, input tensors for the neural network; and
receiving output tensors regarding the upsampling number of frames from the neural network.
11 . A method for encoding video data, comprising:
compressing a frame sequence; and
encoding a supplemental enhancement information (SEI) message for the frame sequence,
wherein the SEI message indicates whether to perform temporal upsampling to the frame sequence by using a neural network;
wherein the SEI message comprises a first indicator for indicating an upsampling number, and a second indicator for indicating an upsampling strength describing a relationship between the upsampling number and a referencing range, and
wherein the referencing range is determined based on the first indicator and the second indicator.
12 . The method according to claim 11 , wherein the SEI message further indicates how to perform temporal upsampling to the frame sequence by using the neural network.
13 . The method according to claim 12 , wherein the neural network interpolates the upsampling number of frames between two adjacent frames of the frame sequence by referencing frames within the referencing range.
14 . The method according to claim 11 , wherein the SEI message further comprises a third indicator for indicating the referencing range, and the referencing range is determined based on the third indicator when the second indicator is equal to zero.
15 . The method according to claim 13 , wherein the SEI message comprises a third indicator for indicating the referencing range.
16 . A method for signaling a bitstream, the method comprising:
receiving a frame sequence;
encoding the frame sequence by:
encoding a supplemental enhancement information (SEI) message for the frame sequence, wherein the SEI message indicates whether to perform temporal upsampling to the frame sequence by using a neural network; wherein the SEI message comprises a first indicator for indicating an upsampling number, and a second indicator for indicating an upsampling strength describing a relationship between the upsampling number and a referencing range, and wherein the referencing range is determined based on the first indicator and the second indicator; and
signaling a bitstream that is generated based on the encoding.
17 . The method according to claim 16 , wherein the SEI message further indicates how to perform temporal upsampling to the frame sequence by using the neural network.
18 . The method according to claim 17 , wherein the neural network interpolates the upsampling number of frames between two adjacent frames of the frame sequence by referencing frames within the referencing range.
19 . The method according to claim 16 , wherein the SEI message further comprises a third indicator for indicating the referencing range, and the referencing range is determined based on the third indicator when the second indicator is equal to zero.