Scaled spatiotemporal transformers for text-to-video synthesis

1 . A text-to-video device, comprising:

a far-reaching interleaved transformer (FIT) block configured to learn a compressed representation of video input having text using a set of learnable latent tokens, wherein the FIT block includes a diffusion framework and joint spatiotemporal modeling using a two-stage cascaded model, the FIT block configured to:

produce a sequence of patch tokens from the text that are divided into groups;

instantiate the set of latent tokens and apply a sequence of computational blocks, wherein each of the computational blocks are configured to:

perform a cross attention read operation between the latent tokens and conditioning signals;

perform a groupwise read cross attention operation between the latent tokens and the patch tokens of corresponding said groups to compress patch information;

apply a series of self attention operations to the latent tokens; and

perform a groupwise write cross attention operation that decompresses information in the latent tokens to update the patch tokens; and

project the patch tokens to generate video frames.

2 . The text-to-video device of claim 1 , wherein one stage of the two-stage cascaded model is configured to focus on motion modeling and the other stage is configured to focus on high-frequency details.

3 . The text-to-video device of claim 2 , wherein the diffusion framework is configured to perform a diffusion process where a process signal-to-noise ratio (SNR) is configured to be controlled through scaling of an input signal.

4 . The text-to-video device of claim 2 , wherein the groups of patch tokens span both temporal and spatial dimensions.

5 . The text-to-video device of claim 4 , wherein the computational blocks are configured to produce the patch tokens by considering three dimensional patches of size T×H×W spanning both the spatial and temporal dimensions, where T represents the temporal dimension, H represents height in the spatial dimension, and W represents width in the spatial dimension.

6 . The text-to-video device of claim 1 , wherein the computational blocks are configured to apply conditioning on the set of latent tokens through a cross attention read operation.

7 . The text-to-video device of claim 1 , wherein the FIT block further comprise a text encoder configured to perform text conditioning and extract the text from the video input.

8 . The text-to-video device of claim 7 , wherein the text conditioning is configured to use conditioning information represented by a sequence of conditioning tokens to control a generation process, wherein the conditioning tokens represent noise, framerate, and original resolution of the video input.

9 . The text-to-video device of claim 8 , wherein the conditioning tokens are configured to support variable video framerates and large differences in resolution and aspect ratios in training data.

10 . The text-to-video device of claim 1 , wherein the FIT block comprises a feed forward module after each cross attention read or write operation.

11 . A method of using a text-to-video device comprising a far-reaching interleaved transformer (FIT) block configured to learn a compressed representation of video input having text using a set of learnable latent tokens, wherein the FIT block includes a diffusion framework and joint spatiotemporal modeling using a two-stage cascaded model, the method comprising the FIT block:

producing a sequence of patch tokens from the text that are divided into groups;

instantiating the set of latent tokens and applying a sequence of computational blocks, wherein each of the computational blocks:

perform a cross attention read operation between the latent tokens and conditioning signals;

perform a groupwise read cross attention operation between the latent tokens and the patch tokens of corresponding said groups to compress patch information;

apply a series of self attention operations to the latent tokens; and

perform a groupwise write cross attention operation that decompresses information in the latent tokens to update the patch tokens; and

projecting the patch tokens to generate video frames.

12 . The method of claim 11 , wherein one stage of the two-stage cascaded model focuses on motion modeling and the other stage focuses on high-frequency details.

13 . The method of claim 12 , wherein the groups of patch tokens span both temporal and spatial dimensions.

14 . The method of claim 13 , wherein the computational blocks produce the patch tokens by considering three dimensional patches of size T×H×W spanning both the spatial and temporal dimensions, where T represents the temporal dimension, H represents height in the spatial dimension, and W represents width in the spatial dimension.

15 . The method of claim 11 , wherein the computational blocks apply conditioning on the set of latent tokens through a cross attention read operation.

16 . The method of claim 11 , wherein the FIT block further comprises a text encoder performing text conditioning and extract the text from the video input.

17 . The method of claim 16 , wherein the text conditioning uses conditioning information represented by a sequence of conditioning tokens to control a generation process, wherein the conditioning tokens represent noise, framerate, and original resolution of the video input.

18 . The method of claim 17 , wherein the conditioning tokens support variable video framerates and large differences in resolution and aspect ratios in training data.

19 . The method of claim 11 , wherein the FIT block comprises a feed forward module after each cross attention read or write operation.

20 . A non-transitory computer readable medium storing program code, which when executed, is operative to cause a text-to-video device comprising a far-reaching interleaved transformer (FIT) block configured to learn a compressed representation of video input having text using a set of learnable latent tokens, wherein the FIT block includes a diffusion framework and joint spatiotemporal modeling using a two-stage cascaded model, to perform:

producing a sequence of patch tokens from the text that are divided into groups;

instantiating the set of latent tokens and applying a sequence of computational blocks, wherein each of the computational blocks:

perform a cross attention read operation between the latent tokens and conditioning signals;

perform a groupwise read cross attention operation between the latent tokens and the patch tokens of corresponding said groups to compress patch information;

apply a series of self attention operations to the latent tokens; and

perform a groupwise write cross attention operation that decompresses information in the latent tokens to update the patch tokens; and

projecting the patch tokens to generate video frames.