Robust vision transformers

1 . A system, comprising:

at least one processor;

at least one memory comprising instructions that, in response to execution by the at least one processor, cause the system to at least:

receive one or more input tokens of an input at one or more encoders, the one or more encoders comprising at least one self-attention portion and at least one channel self-attention portion, wherein the one or more input tokens correspond to respective portions of the input;

aggregate, at the at least one self-attention portion, the one or more input tokens;

generate one or more feature representations of the input tokens based, at least in part, on the aggregation of the one or more input tokens;

aggregate, at the at least one channel self-attention portion, the one or more feature representations of the input tokens, wherein the channel self-attention portion computes an attention matrix along a channel dimension of the input, wherein the one or more feature representations are aggregated based, at least in part, on a correlation value, between feature channels, that exceeds a threshold value;

generate an inference about the input based, at least in part, on the aggregated one or more feature representations; and

generate an output based, at least in part, on the inference.

2 . The system of claim 1 , wherein the output is to exhibit at least a threshold level of resistance to corruptions of the input.

3 . The system of claim 1 , wherein the at least one channel self-attention portion is to generate the one or more feature representations of the input tokens by performing a feature transformation based, at least in part, on a multilayer perceptron block, wherein the multilayer perceptron block is to combine channel information across the one or more feature representations.

4 . The system of claim 1 , wherein the one or more encoders are to aggregate feature channels associated with high correlation values and filter out feature channels associated with low correlation values.

5 . The system of claim 1 , wherein the at least one channel self-attention portion is to compute the attention matrix along the channel dimension of the input based, at least in part, on a feature correlation among the one or more feature representations.

6 . A method, comprising:

receiving one or more input tokens of an input image at one or more encoders, the one or more encoders comprising at least one self-attention portion and at least one channel self-attention portion;

aggregating, by the at least one self-attention portion, the one or more input tokens;

generating one or more feature representations of the input tokens based, at least in part, on the aggregated one or more input tokens;

aggregating, by the at least one channel self-attention portion, the one or more feature representations of the input tokens by combining the one or more feature representations of the input tokens with an attention matrix indicating an amount of correlation between different channels associated with the input image;

generating an inference about the input image based, at least in part, on the aggregated one or more feature representations; and

generating an output based, at least in part, on the inference.

7 . The method of claim 6 , wherein a multilayer perceptron block of the at least one channel self-attention portion is to generate the one or more feature representations of the input tokens using channel information from the input tokens.

8 . The method of claim 6 , further comprising generating the output by at least:

minimizing mutual information between a latent feature representation and a target class label; and

maximizing mutual information between the latent feature representation and input data.

9 . The method of claim 6 , further comprising:

using the one or more encoders to generate one or more outputs indicative of a detected image.

10 . The method of claim 6 , further comprising:

aggregating, using the one or more encoders, feature channels associated with high correlation values; and

isolating, using the one or more encoders, other feature channels associated with low correlation values.

11 . The method of claim 6 , further comprising:

using the one or more encoders to generate one or more outputs indicative of semantic segmentation data.

12 . The method of claim 6 , further comprising:

training the one or more encoders to detect an object in an image based at least in part on multi-scale training using a plurality of training images with a plurality of resolutions.

13 . The method of claim 6 , further comprising:

computing, with the at least one channel self-attention portion, the attention matrix along a channel dimension of the input image based, at least in part, on feature covariance among the one or more feature representations.

14 . The method of claim 6 , wherein the at least one channel self-attention portion aggregates the one or more feature representations of the input tokens based, at least in part, on feature channels with correlation values that exceed a threshold value.

15 . A non-transitory computer-readable medium comprising instructions that, in response to execution by a computing device, cause the computing device to at least:

receive a plurality of input tokens of an input image at one or more layers of a transformer, the one or more layers comprising at least one self-attention portion and at least one channel self-attention portion;

aggregate, at the at least one self-attention portion, the plurality of input tokens;

generate one or more feature representations of the plurality of input tokens based, at least in part, on the aggregation of the plurality of input tokens;

aggregate, at the at least one channel self-attention portion, the one or more feature representations using an attention matrix indicating an amount of correlation between different channels associated with the input image;

generate an inference about the input image based, at least in part, on the aggregated one or more feature representations; and

generate an output based, at least in part, on the inference.

16 . The non-transitory computer-readable medium of claim 15 , wherein the instructions, in response to execution by the computing device, further cause the computing device to at least:

minimize mutual information between a latent feature representation and a target class label; and

maximize mutual information between the latent feature representation and input data.

17 . The non-transitory computer-readable medium of claim 15 , wherein the at least one channel self-attention portion aggregates the one or more feature representations based, at least in part, on one or more feature channels, wherein the one or more feature channels comprise values that exceed a correlation threshold.

18 . The non-transitory computer-readable medium of claim 15 , wherein the output exhibits a threshold level of resistance against one or more noise corruptions of the input image.

19 . The non-transitory computer-readable medium of claim 15 , wherein the at least one channel self-attention portion aggregates one or more feature channels associated with a high correlation value and isolates feature channels associated with a low correlation values.

20 . The non-transitory computer-readable medium of claim 15 , wherein the at least one channel self-attention portion is to aggregate one or more feature channels by multiplying the one or more feature representations with the attention matrix.