TOKEN PRUNING IN SWIN TRANSFORMER ARCHITECTURES

1 . A method, comprising:

receiving input data into a Swin transformer model;

processing the input data in the Swin transformer model to obtain tokens for a tokenized input;

selecting a pruning method for the input data from:

removing;

packaging; and

merging;

pruning the tokens using a token pruning module, which performs the selected token pruning method; and

outputting pruned data.

2 . The method of claim 1 , wherein the selected pruning method is removing, and removing comprises:

identifying windows into which the tokenized input is divided and a pruning target;

identifying N tokens in each window having a lowest information content, where an area of each window is N greater than an area for each window to meet the pruning target;

removing the N tokens from each window to achieve the pruning target with remaining tokens in each window; and

reintegrating each of the windows from the remaining tokens therein.

3 . The method of claim 2 , wherein the reintegrated window maintains a relative order of the remaining tokens in each window from before pruning to after pruning.

4 . The method of claim 1 , wherein the selected pruning method is packaging, and packaging comprises:

identifying windows into which the tokenized input is divided and a pruning target;

identifying P tokens in each window having a lowest information content, where an area of each window is P−1 greater than an area for each window to meet the pruning target;

removing the P tokens from each window;

combining the P removed tokens to produce a packaged token for each window from the P removed tokens from that window; and

reintegrating each of the windows from remaining tokens therein and the packaged token.

5 . The method of claim 4 , wherein the packaged token is reintegrated into the each of the windows in a shared location across the windows.

6 . The method of claim 1 , wherein the selected pruning method is merging, and merging comprises:

identifying initial windows into which the tokenized input is divided and a pruning target;

identifying D 1 tokens in each initial window, excluding those tokens located in a first row of each initial window, having a lowest information content;

merging each of the D 1 tokens in each initial window into another token in that initial window in a vertical direction to transform each initial window into a corresponding intermediate window having a height that is D 1 smaller than a height of the initial window;

identifying D 2 tokens in each intermediate window, excluding those tokens located in a first column of each intermediate window, having a lowest information content;

merging each of the D 2 tokens in each intermediate window into another token in that intermediate window in a horizontal direction to transform each intermediate window into a corresponding spatially complete window having a width that is D 2 smaller than a width of the initial window and the intermediate window.

7 . The method of claim 6 , further comprising:

identifying D 3 tokens in each spatially complete window, excluding those tokens located in a first temporal plane of each spatially complete window, having a lowest information content;

merging each of the D 3 tokens in each spatially complete window into another token in that spatially complete window in a temporal direction to transform each spatially complete window into a tempo-spatially complete window having a time that is D 3 shorter than a time of the initial window, the intermediate window, and the spatially complete window.

8 . A system, comprising:

a processor; and

a memory, storing instructions that, when executed by the processor, perform operations that include:

receiving input data into a Swin transformer model;

processing the input data in the Swin transformer model to obtain tokens for a tokenized input;

selecting a pruning method for the input data from:

removing;

packaging; and

merging;

pruning the tokens using a token pruning module, which performs the selected token pruning method; and

outputting pruned data.

9 . The system of claim 8 , wherein the selected pruning method is removing, and removing comprises:

identifying windows into which the tokenized input is divided and a pruning target;

identifying N tokens in each window having a lowest information content, where an area of each window is N greater than an area for each window to meet the pruning target;

removing the N tokens from each window to achieve the pruning target with remaining tokens in each window; and

reintegrating each of the windows from the remaining tokens therein.

10 . The system of claim 9 , wherein the reintegrated window maintains a relative order of the remaining tokens in each window from before pruning to after pruning.

11 . The system of claim 8 , wherein the selected pruning method is packaging, and packaging comprises:

identifying windows into which the tokenized input is divided and a pruning target;

identifying P tokens in each window having a lowest information content, where an area of each window is P−1 greater than an area for each window to meet the pruning target;

removing the P tokens from each window;

combining the P removed tokens to produce a packaged token for each window from the P removed tokens from that window; and

reintegrating each of the windows from remaining tokens therein and the packaged token.

12 . The system of claim 11 , wherein the packaged token is reintegrated into the each of the windows in a shared location across the windows.

13 . The system of claim 8 , wherein the selected pruning method is merging, and merging comprises:

identifying initial windows into which the tokenized input is divided and a pruning target;

identifying D 1 tokens in each initial window, excluding those tokens located in a first row of each initial window, having a lowest information content;

merging each of the D 1 tokens in each initial window into another token in that initial window in a vertical direction to transform each initial window into a corresponding intermediate window having a height that is D 1 smaller than a height of the initial window;

identifying D 2 tokens in each intermediate window, excluding those tokens located in a first column of each intermediate window, having a lowest information content;

merging each of the D 2 tokens in each intermediate window into another token in that intermediate window in a horizontal direction to transform each intermediate window into a corresponding spatially complete window having a width that is D 2 smaller than a width of the initial window and the intermediate window.

14 . The system of claim 13 , further comprising:

identifying D 3 tokens in each spatially complete window, excluding those tokens located in a first temporal plane of each spatially complete window, having a lowest information content;

merging each of the D 3 tokens in each spatially complete window into another token in that spatially complete window in a temporal direction to transform each spatially complete window into a tempo-spatially complete window having a time that is D 3 shorter than a time of the initial windows, the intermediate window, and the spatially complete window.

15 . A non-transitory computer readable storage device, including instructions that, when executed by a processor, perform operations that include:

receiving input data into a Swin transformer model;

processing the input data in the Swin transformer model to obtain tokens for a tokenized input;

selecting a pruning method for the input data from:

removing;

packaging; and

merging;

pruning the tokens using a token pruning module, which performs the selected token pruning method; and

outputting pruned data.

16 . The device of claim 15 , wherein the selected pruning method is removing, and removing comprises:

identifying windows into which the tokenized input is divided and a pruning target;

identifying N tokens in each window having a lowest information content, where an area of each window is N greater than an area for each window to meet the pruning target;

removing the N tokens from each window to achieve the pruning target with remaining tokens in each window; and

reintegrating each of the windows from the remaining tokens therein.

17 . The device of claim 16 , wherein the reintegrated window maintains a relative order of the remaining tokens in each window from before pruning to after pruning.

18 . The device of claim 15 , wherein the selected pruning method is packaging, and packaging comprises:

identifying windows into which the tokenized input is divided and a pruning target;

identifying P tokens in each window having a lowest information content, where an area of each window is P−1 greater than an area for each window to meet the pruning target;

removing the P tokens from each window;

combining the P removed tokens to produce a packaged token for each window from the P removed tokens from that window; and

reintegrating each of the windows from remaining tokens therein and the packaged token.

19 . The device of claim 15 , wherein the selected pruning method is merging, and merging comprises:

identifying initial windows into which the tokenized input is divided and a pruning target;

identifying D 1 tokens in each initial window, excluding those tokens located in a first row of each initial window, having a lowest information content;

merging each of the D 1 tokens in each initial window into another token in that initial window in a vertical direction to transform each initial window into a corresponding intermediate window having a height that is D 1 smaller than a height of the initial window;

identifying D 2 tokens in each intermediate window, excluding those tokens located in a first column of each intermediate window, having a lowest information content;

merging each of the D 2 tokens in each intermediate window into another token in that intermediate window in a horizontal direction to transform each intermediate window into a corresponding spatially complete window having a width that is D 2 smaller than a width of the initial window and the intermediate window.

20 . The device of claim 19 , further comprising:

identifying D 3 tokens in each spatially complete window, excluding those tokens located in a first temporal plane of each spatially complete window, having a lowest information content;

merging each of the Ds tokens in each spatially complete window into another token in that spatially complete window in a temporal direction to transform each spatially complete window into a tempo-spatially complete window having a time that is Ds shorter than a time of the initial window, the intermediate window, and the spatially complete window.