Global context vision transformer
Vision transformers are deep learning models that employ a self-attention mechanism to obtain feature representations for an input image. To date, the configuration of vision transformers has limited the self-attention computation to a local window of the input image, such that short-range dependencies are modeled in the output. The present disclosure provides a vision transformer that captures global context, and that is therefore able to model long-range dependencies in its output.
1 . A method, comprising:
at a device:
processing an input image through at least one stage of a vision transformer to obtain feature representations for the input image, each stage in the at least one stage including:
a global token generator that, only once during the stage, extracts global features from the input image and generates from the global features repeating global query tokens to spatially match with local tokens of each local window of a plurality of local windows within the input image such that each of the repeating global query tokens corresponds with a different one of the local tokens,
a global self-attention module that uses the repeating global query tokens to compute global-self attention for each local window of the plurality of local windows within the input image, and
a local self-attention module that computes local-self attention for each local window of the plurality of local windows; and
outputting the feature representations.
2 . The method of claim 1 , wherein the input image is apportioned into the plurality of local windows.
3 . The method of claim 1 , wherein for each local window of the plurality of local windows, the plurality of image patches within the local window overlap.
4 . The method of claim 1 , wherein the local self-attention module computes local query, key, and value tokens for each image patch of the plurality of image patches within the local window, and wherein the local interactions are captured using computations applied to the local query, key, and value tokens.
5 . The method of claim 1 , wherein the global features are extracted from an entirety of the input image.
6 . The method of claim 1 , wherein a global features are extracted from a feature map created for an entirety of the input image.
7 . The method of claim 1 , wherein each stage of the at least one stage of the vision transformer computes local and global self-attention, per local window of the plurality of local windows.
8 . The method of claim 1 , wherein a spatial resolution is decreased after one or more stages in the at least one stage.
9 . The method of claim 8 , wherein the spatial resolution is decreased by a downsampling block of the vision transformer.
10 . The method of claim 9 , wherein the downsampling block includes a Fused-MBConv block that provides inductive bias and modeling of inter-channel dependencies when decreasing the spatial resolution.
11 . The method of claim 1 , wherein the feature representations are output as embeddings for the input image.
12 . The method of claim 11 , wherein the feature representations are output to one or more further processing blocks of the vision transformer to create the embeddings.
13 . The method of claim 12 , wherein the further processing blocks include average pooling and linear layers.
14 . The method of claim 1 , wherein the feature representations are output to a computer vision-related downstream task.
15 . The method of claim 14 , wherein the computer vision-related downstream task performs one of:
image classification,
object detection,
instance segmentation, or
semantic segmentation.
16 . The method of claim 1 , wherein the input image is processed through a plurality of stages, and wherein each stage in the plurality of stages includes the global token generator, the global self-attention module and the local self-attention module.
17 . The method of claim 16 , wherein the plurality of stages are sequential.
18 . A non-transitory computer-readable media storing computer instructions which when executed by one or more processors of a device cause the device to:
process an input image through at least one stage of a vision transformer to obtain feature representations for the input image, each stage in the at least one stage including:
a global token generator that, only once during the stage, extracts global features from the input image and generates from the global features repeating global query tokens to spatially match with local tokens of each local window of a plurality of local windows within the input image such that each of the repeating global query tokens corresponds with a different one of the local tokens,
a global self-attention module that uses the repeating global query tokens to compute global-self attention for each local window of the plurality of local windows within the input image, and
a local self-attention module that computes local-self attention for each local window of the plurality of local windows; and
output the feature representations.
19 . A system, comprising:
a non-transitory memory storage of a receiving device comprising instructions; and
one or more processors of the receiving device in communication with the memory, wherein the one or more processors execute the instructions to:
process an input image through at least one stage of a vision transformer to obtain feature representations for the input image, each stage in the at least one stage including:
a global token generator that, only once during the stage, extracts global features from the input image and generates from the global features repeating global query tokens to spatially match with local tokens of each local window of a plurality of local windows within the input image such that each of the repeating global query tokens corresponds with a different one of the local tokens,
a global self-attention module that uses the repeating global query tokens to compute global-self attention for each local window of the plurality of local windows within the input image, and
a local self-attention module that computes local-self attention for each local window of the plurality of local windows; and
output the feature representations.
20 . A method, comprising:
at a device:
processing an input image through at least one stage of a vision transformer to obtain feature representations for the input image, each stage in the at least one stage including:
a global token generator that, only once during the stage, extracts global features from the input image and generates from the global features repeating global query tokens to spatially match with local tokens of each local window of a plurality of local windows within the input image such that each of the repeating global query tokens corresponds with a different one of the local tokens,
a global self-attention module that uses the repeating global query tokens to compute global-self attention for each local window of the plurality of local windows within the input image; and
outputting the feature representations.
21 . The method of claim 1 , wherein the global-self attention is computed for each local window of the plurality of local windows within the input image by computing interaction of each of the repeating global tokens with the corresponding one of the local tokens of the local window.
22 . The method of claim 1 , wherein the local-self attention is computed for each local window of the plurality of local windows by computing local interaction of the local tokens of the local window.