Multi-scale transformer for image analysis
The technology employs a patch-based multi-scale Transformer ( 300 ) that is usable with various imaging applications. This avoids constraints on image fixed input size and predicts the quality effectively on a native resolution image. A native resolution image ( 304 ) is transformed into a multi-scale representation ( 302 ), enabling the Transformer's self-attention mechanism to capture information on both fine-grained detailed patches and coarse-grained global patches. Spatial embedding ( 316 ) is employed to map patch positions to a fixed grid, in which patch locations at each scale are hashed to the same grid. A separate scale embedding ( 318 ) is employed to distinguish patches coming from different scales in the multiscale representation. Self-attention ( 508 ) is performed to create a final image representation. In some instances, prior to performing self-attention, the system may prepend a learnable classification token ( 322 ) to the set of input tokens.
1. A method for processing imagery, the method comprising:
constructing, by one or more processors, a multi-scale representation of a native resolution image, the multi-scale representation including the native resolution image and a set of aspect ratio preserving resized variants;
encoding, by the one or more processors, a corresponding spatial embedding for each patch associated with a respective region of either the native resolution image or one of the set of aspect ratio preserving resized variants, thereby forming a set of spatially encoded patches;
applying, by the one or more processors, a set of scale embeddings to the set of spatially encoded patches to capture scale information associated with the native resolution image and the set of aspect ratio resized variants, thereby forming a set of input tokens; and
performing, by the one or more processors according to a transformer encoder module, self-attention on the set of input tokens to create a final image representation.
2. The method of claim 1 , wherein the final image representation corresponds to a predicted quality score of the native resolution image.
3. The method of claim 1 , wherein constructing the multi-scale representation includes splitting each of the native resolution image and each aspect ratio preserving resized variant into fixed-size patches, wherein each patch represents a distinct region of either the native resolution image or one of the aspect ratio preserving resized variants.
4. The method of claim 1 , wherein encoding the corresponding spatial embedding includes hashing a patch position for each patch within a grid of learnable embeddings.
5. The method of claim 1 , wherein each aspect ratio preserving resized variant is formed so that an aspect ratio of that variant is sized according to a longer side of the native resolution image.
6. The method of claim 1 , wherein the multi-scale representation includes the native resolution image with height H, width W and one or more channels C, and the set of aspect ratio preserving resized variants is derived using a Gaussian kernel, wherein each channel represents a color component of the native resolution image.
7. The method of claim 1 , wherein, to align the set of spatially encoded patches across scales, patch locations from all scales are mapped to a same grid.
8. The method of claim 1 , further comprising, prior to self-attention, prepending a learnable classification token to the set of input tokens.
9. The method of claim 1 , wherein performing the self-attention on the set of input tokens includes performing multi-head self-attention on the set of input tokens, including:
performing a first normalization on the set of input tokens to generate a set of first normalizations;
performing a multi-head self-attention operation on the set of first normalizations;
performing a second normalization on output of the multi-head self-attention operation to generate a second normalization; and
performing a multi-layer perceptron process on the second normalization to create a final image representation.
10. The method of claim 9 , wherein the second normalization is performed on the output of the multi-head self-attention operation added to a residual connection from the set of input tokens.
11. The method of claim 10 , wherein the final image representation is obtained by adding input from the second normalization to output from the multi-layer perceptron process.
12. The method of claim 1 , further comprising pretraining a transformer of the transformer encoder module on a selected image dataset.
13. The method of claim 1 , wherein a size of the patches is selected based on an average resolution across the native resolution image and the set of aspect ratio preserving resized variants.
14. The method of claim 13 , wherein the size of the patches is further based on a size of a grid of learnable embeddings to which the patches are mapped.
15. The method of claim 1 , wherein encoding the spatial embedding for each patch comprises employing a convolutional neural network with a fully connected layer.
16. An image processing system, comprising:
memory configured to store imagery; and
one or more processors operatively coupled to the memory, the one or more processors being configured to
construct a multi-scale representation of a native resolution image, the multi-scale representation including the native resolution image and a set of aspect ratio preserving resized variants;
encode a corresponding spatial embedding for each patch associated with a respective region of either the native resolution image or one of the set of aspect ratio preserving resized variants, to form a set of spatially encoded patches;
apply a set of scale embeddings to the set of spatially encoded patches to capture scale information associated with the native resolution image and the set of aspect ratio resized variants, to form a set of input tokens; and
employ a transformer encoder module to perform self-attention on the set of input tokens to create a final image representation.
17. The image processing system of claim 16 , wherein the one or more processors are further configured to store in the memory at least one of the final image representation, the native resolution image, or the set of aspect ratio preserving resized variants.
18. The image processing system of claim 17 , wherein performance of the self-attention on the set of input tokens includes performance of multi-head self-attention on the set of input tokens, including:
performance of a first normalization on the set of input tokens to generate a set of first normalizations;
performance of a multi-head self-attention operation on the set of first normalizations;
performance of a second normalization on output of the multi-head self-attention operation to generate a second normalization; and
performance of a multi-layer perceptron process on the second normalization to create a final image representation.
19. The image processing system of claim 16 , wherein the final image representation corresponds to a predicted quality score of the native resolution image.
20. The image processing system of claim 16 , wherein construction of the multi-scale representation includes splitting each of the native resolution image and each aspect ratio preserving resized variant into fixed-size patches, wherein each patch represents a distinct region of either the native resolution image or one of the aspect ratio preserving resized variants.
21. The image processing system of claim 16 , wherein encoding the corresponding spatial embedding includes hashing a patch position for each patch within a grid of learnable embeddings.
22. The image processing system of claim 16 , wherein the one or more processors are further configured, prior to performing self-attention, to prepend a learnable classification token to the set of input tokens.