Method and apparatus for classifying images using an artificial intelligence model
An apparatus for performing image processing, may include at least one processor configured to: input an image to a vision transformer comprising a plurality of encoders that correspond to at least one fixed encoder and a plurality of adaptive encoders; process the image via the at least one fixed encoder to obtain image representations; determine one or more layers of the plurality of adaptive encoders to drop, by inputting the image representations to a policy network configured to determine layer dropout actions for the plurality of adaptive encoders; and obtain a class of the input image using remaining layers of the plurality of adaptive encoders other than the dropped one or more layers.
1. An apparatus for performing image processing, the apparatus comprising:
a memory storing instructions; and
at least one processor configured to execute the instructions to:
input an image to a vision transformer comprising a plurality of encoders that correspond to at least one fixed encoder and a plurality of adaptive encoders;
process the image via the at least one fixed encoder to obtain image representations;
determine one or more layers of the plurality of adaptive encoders to drop, by inputting the image representations to a policy network configured to determine layer dropout actions for the plurality of adaptive encoders; and
obtain a class of the input image using remaining layers of the plurality of adaptive encoders other than the dropped one or more layers.
2. The apparatus of claim 1 , wherein each of the plurality of encoders comprises a multi-head self-attention (MSA) layer and a multilayer perceptron (MLP) layer.
3. The apparatus of claim 1 , wherein the layer dropout actions indicate whether each multi-head self-attention (MSA) layer and each multilayer perceptron (MLP) layer included in the plurality of adaptive encoders is dropped or not.
4. The apparatus of claim 1 , wherein the policy network comprises a first policy network configured to determine whether to drop one or more multi-head self-attention (MSA) layers, and a second policy network configured to determine whether to drop one or more multilayer perceptron (MLP) layers.
5. The apparatus of claim 4 , wherein the first policy network is configured to receive, as input, the image representations that are output from the at least one fixed encoder of the vision transformer, and output the layer dropout actions for each MSA layer of the plurality of adaptive encoders.
6. The apparatus of claim 5 , wherein the second policy network is further configured to receive, as input, the image representations and the layer dropout actions for each MSA layer, and output the layer dropout actions for each MLP layer of the plurality of adaptive encoders.
7. The apparatus of claim 6 , wherein the second policy network comprises a dense layer configured to receive, as input, a concatenation of the image representations and the layer dropout actions for each MSA layer.
8. The apparatus of claim 1 , wherein the policy network is configured to receive a reward that is calculated based on a number of the dropped one or more layers, and image classification prediction accuracy of the vision transformer.
9. The apparatus of claim 8 , wherein the at least one processor is configured to execute the instructions to:
calculate the reward using a reward function that increases the reward as the number of the dropped one or more layers increases and the image classification prediction accuracy increase.
10. A method of performing image processing, the method being performed by at least one processor, and the method comprising:
inputting an image to a vision transformer comprising a plurality of encoders that correspond to at least one fixed encoder and a plurality of adaptive encoders;
processing the image via the at least one fixed encoder to obtain image representations;
determining one or more layers of the plurality of adaptive encoders to drop, by inputting the image representations to a policy network configured to determine layer dropout actions for the plurality of adaptive encoders; and
obtaining a class of the input image using remaining layers of the plurality of adaptive encoders other than the dropped one or more layers.
11. The method of claim 10 , wherein each of the plurality of encoders comprises a multi-head self-attention (MSA) layer and a multilayer perceptron (MLP) layer.
12. The method of claim 10 , wherein the layer dropout actions indicate whether each multi-head self-attention (MSA) layer and each multilayer perceptron (MLP) layer included in the plurality of adaptive encoders is dropped or not.
13. The method of claim 10 , wherein the determining the one or more layers of the plurality of adaptive encoders to drop, comprises:
determining whether to drop one or more multi-head self-attention (MSA) layers, via a first policy network; and
determining whether to drop one or more multilayer perceptron (MLP) layers, via a second policy network.
14. The method of claim 13 , wherein the determining whether to drop the one or more multi-head self-attention (MSA) layers, comprises:
inputting the image representations that are output from the at least one fixed encoder of the vision transformer, to the first policy network; and
outputting the layer dropout actions for each MSA layer of the plurality of adaptive encoders, from the first policy network.
15. The method of claim 14 , wherein the determining whether to drop the one or more multilayer perceptron (MLP) layers, comprises:
inputting, to the second policy network, the image representations and the layer dropout actions for each MSA layer; and
outputting the layer dropout actions for each MLP layer of the plurality of adaptive encoders, from the second policy network.
16. The method of claim 15 , further comprising:
concatenating the image representations and the layer dropout actions for each MSA layer; and
inputting a concatenation of the image representations and the layer dropout actions for each MSA layer, to a dense layer of the second policy network.
17. The method of claim 10 , wherein the policy network is trained using a reward function that calculates a reward based on a number of the dropped one or more layers, and image classification prediction accuracy of the vision transformer.
18. The method of claim 17 , wherein the reward function increases the reward as the number of the dropped one or more layers increases and the image classification prediction accuracy increase.
19. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to:
input an image to a vision transformer comprising a plurality of encoders that correspond to at least one fixed encoder and a plurality of adaptive encoders;
process the image via the at least one fixed encoder to obtain image representations;
determine one or more of multi-head self-attention (MSA) layers and multilayer perceptron (MLP) layers of the plurality of adaptive encoders to drop, by inputting the image representations to a policy network configured to determine layer dropout actions for the plurality of adaptive encoders; and
obtain a class of the input image using remaining layers of the plurality of adaptive encoders other than the dropped one or more layers.
20. The non-transitory computer-readable storage medium of claim 19 , wherein the policy network is trained using a reward function that increases a reward in direct proportion to a number of the dropped one or more layers and image classification prediction accuracy of the vision transformer.