Unified space-time interpolation of video information
A technique is described herein for temporally and spatially interpolating input video information, to produce output video information having a higher frame rate and a higher resolution compared to that exhibited by the input video information. The technique generates feature information based on plural frames of the input video information. The technique then produces the output video information based on the feature information using an architecture having, in order, a multi-stage encoding operation, a query-generating operation, and a multi-stage decoding operation. Each encoding stage produces an instance of encoder attention information that expresses identified relations across the plural frames of the input video information. Each decoding stage operates on an instance of encoder attention information produced by a corresponding encoding stage. The transformer architecture is compact and is capable of interpolating the input video information in real time.
1. A method for interpolating video information, comprising:
obtaining input video information having a given first number of plural frames, each frame in the input video information having a given first spatial resolution;
generating feature information based the input video information;
encoding the feature information in a pipeline having plural encoding stages that operate at different respective resolutions, to produce plural instances of encoder attention information and plural instances of encoder output information, each instance of the encoder attention information expressing identified relations across the plural frames of the input video information;
producing a query based on an instance of encoder output information produced by a last encoding stage of the plural encoding stages;
decoding the query in a pipeline having plural decoding stages that operate at different respective resolutions, to produce plural instances of decoder output information, each decoding stage that has a preceding decoding stage receiving an instance of decoding input information produced by the preceding decoding stage, and each particular decoding stage operating on an instance of encoder attention information produced by a particular encoding stage that has a same resolution level as the particular decoding stage; and
producing output video information based on decoder output information produced by a last decoding stage of the plural decoding stages,
said producing including:
performing a reconstruction operation on the decoder output information produced by the last decoding stage, to produce reconstructed information;
interpolating the input video information to produce interpolated video information; and
combining the reconstructed information with the interpolated video information to produce the output video information,
the output video information having a second number of frames that is higher than the first number of frames in the input video information, and having a second spatial resolution that is higher than the first spatial resolution of the input video information.
2. The method of claim 1 , wherein each given stage in the pipeline of encoding stages and the pipeline of decoding stages produces a particular instance of attention information using a transformer-based neural network.
3. The method of claim 2 , wherein the transformer-based neural network performs a first kind of attention operation that involves partitioning first-attention-operation input information into individual windows, and generating window-specific attention information for the individual windows.
4. The method of claim 3 , wherein the transformer-based neural network also performs a second kind of attention operation that involves partitioning second-attention-operation input information into individual windows, and generating window-specific attention information for the individual windows produced by the second kind of attention operation, the individual windows in the second kind of attention operation being shifted relative to the individual windows in the first kind of attention operation.
5. The method of claim 2 , wherein the given stage is a given encoding stage, and wherein the given encoding stage also performs a down-sampling operation on the particular instance of attention information.
6. The method of claim 2 , wherein the given stage is a given encoding stage, and wherein the particular instance of attention information includes key information and value information that is generated based on feature information obtained from the plural frames of the input video information.
7. The method of claim 2 , wherein the given stage is a given decoding stage, and wherein the given decoding stage also performs an up-sampling operation on the particular instance of attention information.
8. The method of claim 1 , wherein said producing a query involves producing encoder output information for at least one added frame that is not present in the input video information.
9. The method of claim 8 , wherein said producing a query produces the encoder output information for said at least one added frame based on the encoder output information produced by the last encoding stage for frames in the input video information that temporally precede and follow the added frame.
10. A computing system for performing an interpolation operation, comprising:
a memory for storing machine-readable instruction;
a processor that performs operations by executing the machine-readable instructions stored in the memory, the operations including:
obtaining input video information having a given first number of plural frames, each frame in the input video information having a given first spatial resolution;
generating feature information based on the input video information; and
producing output video information based on the feature information using a transformer-based encoding operation, followed by a query-generating operation, followed by a transformer-based decoding operation, followed by an interpolating operation, the transformer-based decoding operation being performed based on encoder attention information produced by the transformer-based encoding operation,
the encoder attention information expressing identified relations across the plural frames of the input video information,
the output video information having a second number of frames that is higher than the first number of frames in the input video information, and having a second spatial resolution that is higher than the first spatial resolution of the input video information,
wherein the transformer-based encoding operation includes a pipeline having plural encoding stages that operate at different respective resolutions, and that produce plural instances of encoder attention information and plural instances of encoder output information,
each instance of the encoder attention information expressing identified relations across the plural frames of the input video information,
wherein the query-generating operation produces a query based on an instance of encoder output information produced by a last encoding stage of the plural encoding stages,
wherein the transformer-based decoding operation decodes the query using a pipeline having plural decoding stages that operate at different respective resolutions, and that produce plural instances of decoder output information, each decoding stage that has a preceding decoding stage receiving an instance of decoder input information produced by the preceding decoding stage, and each particular decoding stage operating on an instance of encoder attention information produced by a particular encoding stage that has a same resolution level as the particular decoding stage,
wherein the interpolating operation interpolates the input video information to produce interpolated video information, and wherein the operations further include combining decoder output information produced by a last decoding stage of the plural decoding stages with the interpolated video information to produce the output video information.
11. The computing system of claim 10 , wherein a given encoding stage performs at least one kind of attention operation and a down-sampling operation.
12. The computing system of claim 10 , wherein a given decoding stage performs at least one kind of attention operation and an up-sampling operation.
13. A non-transitory computer-readable storage medium for storing computer-readable instructions, the computer-readable instructions, when executed by one or more hardware processors, performing a method that comprises:
obtaining input video information having a given first number of plural frames, each frame in the input video information having a given first spatial resolution;
generating feature information based on the input video information; and
producing output video information based on the feature information using a transformer-based encoding operation, followed by a query-generating operation, followed by a transformer-based decoding operation, followed by an interpolating operation, the transformer-based decoding operation being performed based on encoder attention information produced by the transformer-based encoding operation,
the encoder attention information expressing identified relations across the plural frames of the input video information,
the output video information having a second number of frames that is higher than the first number of frames in the input video information, and having a second spatial resolution that is higher than the first spatial resolution of the input video information,
wherein the transformer-based encoding operation includes a pipeline having plural encoding stages that operate at different respective resolutions, and that produce plural instances of encoder attention information and plural instances of encoder output information,
each instance of the encoder attention information expressing identified relations across the plural frames of the input video information,
wherein the query-generating operation produces a query based on an instance of encoder output information produced by a last encoding stage of the plural encoding stages,
wherein the transformer-based decoding operation decodes the query using a pipeline having plural decoding stages that operate at different respective resolutions, and that produce plural instances of decoder output information, each decoding stage that has a preceding decoding stage receiving an instance of decoder input information produced by the preceding decoding stage, and each particular decoding stage operating on an instance of encoder attention information produced by a particular encoding stage that has a same resolution level as the particular decoding stage,
wherein the interpolating operation interpolates the input video information to produce interpolated video information, and wherein the operations further include combining decoder output information produced by a last decoding stage of the plural decoding stages with the interpolated video information to produce the output video information.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the pipeline of encoding stages operate using successively lower resolutions, and wherein the pipeline of decoding stages operate using successively higher resolutions.
15. The non-transitory computer-readable storage medium of claim 13 , wherein each given encoding stage in the pipeline of encoding stages performs a first kind of attention operation that involves partitioning first-attention-operation input information into individual windows, and generating window-specific attention information for the individual windows,
each individual window for which the first kind of attention operation is performed being based on feature information obtained from the plural frames of the input video information.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the given encoding stage also performs at a second kind of attention operation that involves partitioning second-attention-operation input information into individual windows, and generating window-specific attention information for individual windows produced by the second kind of attention operation, the individual windows in the second kind of attention operation being shifted relative to the individual windows in the first kind of attention operation,
each individual window for which the second kind of attention operation is performed being based on feature information obtained from the plural frames of the input video information.
17. The non-transitory computer-readable storage medium of claim 13 , further comprising:
performing a reconstruction operation on the decoder output information produced by the last decoding stage of the plural decoding stages, to produce reconstructed information.