Cascaded local implicit transformer for arbitrary-scale super-resolution

1 . A method, comprising:

receiving a low-resolution (LR) image;

extracting a first feature embedding from the LR image;

performing a first upsampling to the LR image by a first upsampling factor to generate a upsampled image;

receiving a LR coordinate of a pixel within the LR image and a first cell size of the LR coordinate;

generating a first residual image based on the first feature embedding, the LR coordinate, and the first cell size of the LR coordinate using a local implicit image function; and

generating a first high-resolution (HR) image by combining the first residual image and the upsampled image;

wherein the first residual image is generated by:

applying convolution functions to the first feature embedding to extract a local frequency latent embedding at the LR coordinate;

performing upsampling operations to the local frequency latent embedding; and

generating a local frequency embedding;

wherein the local frequency embedding is generated by:

generating a relative positional encoding of the LR coordinate; and

generating the local frequency embedding by multiplying a Fourier transform of the local frequency latent embedding with the relative positional encoding of the LR coordinate.

2 . The method of claim 1 , wherein the first residual image is generated by:

applying convolution functions to the first feature embedding to extract a local query latent embedding, a local key latent embedding and a local value latent embedding at the LR coordinate;

performing upsampling operations to the local query latent embedding, the local key latent embedding, and the local value latent embedding; and

generating a local latent embedding.

3 . The method of claim 2 , wherein the local latent embedding is generated by:

generating an attention matrix by adding an inner product of the local query latent embedding and the local key latent embedding with a relative positional bias of the LR coordinate;

generating a local attention map by normalizing the attention matrix; and

generating the local latent embedding by performing element-wise multiplication of the local value latent embedding and the local attention map.

4 . The method of claim 1 , further comprising:

training the local implicit image function with a first set of upsampling factors; and

training the local implicit image function by alternatively switching between the first set of upsampling factors and a second set of upsampling factors,

wherein upsampling factors in the first set of upsampling factors are smaller than upsampling factors in the second set of upsampling factors.

5 . The method of claim 1 , further comprising:

performing a second upsampling to the first feature embedding by a second upsampling factor to generate a second feature embedding;

generating a second residual image based on the second feature embedding, the LR coordinate, and a second cell size of the LR coordinate using the local implicit image function;

generating a combined residual image by combining the first residual image and the second residual image; and

generating a second HR image by combining the combined residual image and the upsampled image.

6 . An apparatus, comprising circuitry configured to:

receive a low-resolution (LR) image;

extract a first feature embedding from the LR image;

perform a first upsampling to the LR image by a first upsampling factor to generate a upsampled image;

receive a LR coordinate of a pixel within the LR image and a first cell size of the LR coordinate;

generate a first residual image based on the first feature embedding, the LR coordinate, and the first cell size of the LR coordinate by using a local implicit image function; and

generate a first high-resolution (HR) image by combining the first residual image and the upsampled image;

wherein the circuitry is further configured to:

apply convolution functions to the first feature embedding to extract a local frequency latent embedding at the LR coordinate;

perform upsampling operations to the local frequency latent embedding; and

generate a local frequency embedding;

wherein the circuitry is further configured to:

generate a relative positional encoding of the LR coordinate; and

generate the local frequency embedding by multiplying a Fourier transform of the local frequency latent embedding with the relative positional encoding of the LR coordinate.

7 . The apparatus of claim 6 , wherein the circuitry is further configured to:

apply convolution functions to the first feature embedding to extract a local query latent embedding, a local key latent embedding, and a local value latent embedding at the LR coordinate;

perform upsampling operations to the local query latent embedding, the local key latent embedding, and the local value latent embedding; and

generate a local latent embedding.

8 . The apparatus of claim 7 , wherein the circuitry is further configured to:

generate an attention matrix by adding an inner product of the local query latent embedding and the local key latent embedding with a relative positional bias of the LR coordinate;

generate a local attention map by normalizing the attention matrix; and

generate the local latent embedding by performing element-wise multiplication of the local value latent embedding and the local attention map.

9 . The apparatus of claim 6 , wherein the circuitry is further configured to:

train the local implicit image function with a first set of upsampling factors; and

train the local implicit image function by alternatively switching between the first set of upsampling factors and a second set of upsampling factors,

wherein upsampling factors in the first set of upsampling factors are smaller than upsampling factors in the second set of upsampling factors.

10 . The apparatus of claim 8 , wherein the circuitry is further configured to:

perform a second upsampling to the first feature embedding by a second upsampling factor to receive a second feature embedding;

generate a second residual image based on the second feature embedding, the LR coordinate, and a second cell size of the LR coordinate by using the local implicit image function;

generate a combined residual image by combining the first residual image and the second residual image; and

generate a second HR image by combining the combined residual image and the upsampled image.

11 . A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method, the method comprising:

receiving a low-resolution (LR) image;

extracting a first feature embedding from the LR image;

performing a first upsampling to the LR image by a first upsampling factor to generate a upsampled image;

receiving a LR coordinate of a pixel within the LR image and a first cell size of the LR coordinate;

generating a first residual image based on the first feature embedding, the LR coordinate, and the first cell size of the LR coordinate using a local implicit image function; and

generating a first high-resolution (HR) image by combining the first residual image and the upsampled image;

wherein the first residual image is generated by:

applying convolution functions to the first feature embedding to extract a local frequency latent embedding at the LR coordinate;

performing upsampling operations to the local frequency latent embedding; and

generating a local frequency embedding;

wherein the local frequency embedding is generated by:

generating a relative positional encoding of the LR coordinate; and

generating the local frequency embedding by multiplying a Fourier transform of the local frequency latent embedding with the relative positional encoding of the LR coordinate.

12 . The non-transitory computer-readable medium of claim 11 , wherein the method further comprises:

applying convolution functions to the first feature embedding to extract a local query latent embedding, a local key latent embedding, and a local value latent embedding at the LR coordinate;

performing upsampling operations to the local query latent embedding, the local key latent embedding, and the local value latent embedding; and

generating a local latent embedding.

13 . The non-transitory computer-readable medium of claim 12 , wherein the method further comprises:

generating an attention matrix by adding an inner product of the local query latent embedding and the local key latent embedding with a relative positional bias of the LR coordinate;

generating a local attention map by normalizing the attention matrix; and

generating the local latent embedding by performing element-wise multiplication of the local value latent embedding and the local attention map.

14 . The non-transitory computer-readable medium of claim 11 , wherein the method further comprises:

training the local implicit image function with a first set of upsampling factors; and

training the local implicit image function by alternatively switching between the first set of upsampling factors and a second set of upsampling factors,

wherein upsampling factors in the first set of upsampling factors are smaller than upsampling factors in the second set of upsampling factors.

15 . The non-transitory computer-readable medium of claim 11 , wherein the method further comprises:

performing a second upsampling to the first feature embedding by a second upsampling factor to generate a second feature embedding;

generating a second residual image based on the second feature embedding, the LR coordinate, and a second cell size of the LR coordinate using the local implicit image function;

generating a combined residual image by combining the first residual image and the second residual image; and

generating a second HR image by combining the combined residual image and the upsampled image.