Saliency-guided mixup with optimal re-arrangements for efficient data augmentation

1 . A method of performing data augmentation by a device, comprising:

obtaining a plurality of images from a dataset, each image of the plurality of images having a corresponding label;

computing, for each image of the plurality of images, a corresponding saliency map based on a gradient of a full loss function of that image;

selecting, from a subset of arrangements of a plurality of possible arrangements, a rearrangement offset that maximizes an overall saliency of a resulting image combining the plurality of images;

computing, based on the corresponding saliency maps, a two-dimensional mixing mask, wherein the two-dimensional mixing mask includes, for each pixel location, a different mixing ratio determined based on saliency values of the plurality of images at a same pixel location, wherein elements of the two-dimensional mixing mask indicate a saliency ratio between the plurality of images at the same pixel location;

generating, using the rearrangement offset and the two-dimensional mixing mask, a new mixed image by element-wise combining the plurality of images and a new mixed label from corresponding labels of the plurality of images; and

augmenting the dataset with the new mixed image and the new mixed label.

2 . The method of claim 1 , wherein the computing, for each image of the plurality of images, of the corresponding saliency map comprises:

providing that image of the plurality of images to a classifier model;

measuring the full loss function between an output of the classifier model and the corresponding label of that image; and

obtaining the corresponding saliency map by calculating the gradient of the full loss function.

3 . The method of claim 2 , wherein the obtaining of the corresponding saliency map comprises:

normalizing at least one dimension of the gradient of the full loss function; and

applying a Gaussian smoothing parameter to the normalized gradient.

4 . The method of claim 1 , wherein the selecting of the rearrangement offset comprises:

identifying the subset of arrangements from the plurality of possible arrangements, each arrangement of the subset of arrangements indicating a distinct positional translation of the plurality of images, the subset of arrangements comprising a predetermined portion of the plurality of possible arrangements;

calculating overall saliencies of sample images resulting from the distinct positional translations of the plurality of images indicated by the subset of arrangements; and

selecting, as the rearrangement offset, an arrangement from the subset of arrangements corresponding to a maximum overall saliency from among the overall saliencies.

5 . The method of claim 4 , wherein the identifying of the subset of arrangements comprises:

randomly identifying the predetermined portion of the plurality of possible arrangements as the subset of arrangements, the predetermined portion comprising approximately one percent (1%) of the plurality of possible arrangements.

6 . The method of claim 1 , wherein the generating of the new mixed label comprises:

interpolating the corresponding labels of the plurality of images based on the overall saliency of the new mixed image.

7 . The method of claim 1 , wherein:

the plurality of images comprises a first image, the first image having a first corresponding saliency map,

the method further comprises:

training a classifier model with the augmented dataset; and

computing, using the trained classifier model, a new saliency map corresponding to the first image, and

wherein the new saliency map is different from the first corresponding saliency map.

8 . A device for performing data augmentation, comprising:

a memory storing instructions; and

a processor communicatively coupled to the memory, wherein the processor is configured to execute the instructions to:

obtain a plurality of images from a dataset, each image of the plurality of images having a corresponding label;

compute, for each image of the plurality of images, a corresponding saliency map based on a gradient of a full loss function of that image;

select, from a subset of arrangements of a plurality of possible arrangements, a rearrangement offset that maximizes an overall saliency of a resulting image combining the plurality of images;

compute, based on the corresponding saliency maps, a two-dimensional mixing mask, wherein the two-dimensional mixing mask includes, for each pixel location, a different mixing ratio determined based on saliency values of the plurality of images at a same pixel location, wherein elements of the two-dimensional mixing mask indicate a saliency ratio between the plurality of images at the same pixel location;

generate, using the rearrangement offset and the two-dimensional mixing mask, a new mixed image by element-wise combining the plurality of images and a new mixed label from corresponding labels of the plurality of images; and

augment the dataset with the new mixed image and the new mixed label.

9 . The device of claim 8 , the processor is further configured to execute the instructions to:

provide each image of the plurality of images to a classifier model;

measure the full loss function between an output of the classifier model and the corresponding label of each image of the plurality of images; and

obtain the corresponding saliency map by calculating the gradient of the full loss function.

10 . The device of claim 9 , the processor is further configured to execute the instructions to:

normalize at least one dimension of the gradient of the full loss function; and

apply a Gaussian smoothing parameter to the normalized gradient.

11 . The device of claim 8 , the processor is further configured to execute the instructions to:

identify the subset of arrangements from the plurality of possible arrangements, each arrangement of the subset of arrangements indicating a distinct positional translation of the plurality of images, the subset of arrangements comprising a predetermined portion of the plurality of possible arrangements;

calculate overall saliencies of sample images resulting from the distinct positional translations of the plurality of images indicated by the subset of arrangements; and

select, as the rearrangement offset, an arrangement from the subset of arrangements corresponding to a maximum overall saliency from among the overall saliencies.

12 . The device of claim 11 , the processor is further configured to execute the instructions to:

randomly identify the predetermined portion of the plurality of possible arrangements as the subset of arrangements, the predetermined portion comprising approximately one percent (1%) of the plurality of possible arrangements.

13 . The device of claim 8 , the processor is further configured to execute the instructions to:

interpolate the corresponding labels of the plurality of images based on the overall saliency of the new mixed image.

14 . The device of claim 8 , wherein:

the plurality of images comprises a first image, the first image having a first corresponding saliency map,

the processor is further configured to execute the instructions to:

train a classifier model with the augmented dataset; and

compute, using the trained classifier model, a new saliency map corresponding to the first image, and

wherein the new saliency map is different from the first corresponding saliency map.

15 . A non-transitory computer-readable storage medium storing computer-executable instructions for performing data augmentation that, when executed by at least one processor of a device, cause the device to:

obtain a plurality of images from a dataset, each image of the plurality of images having a corresponding label;

compute, for each image of the plurality of images, a corresponding saliency map based on a gradient of a full loss function of that image;

select, from a subset of arrangements of a plurality of possible arrangements, a rearrangement offset that maximizes an overall saliency of a resulting image combining the plurality of images;

compute, based on the corresponding saliency maps, a two-dimensional mixing mask, wherein the two-dimensional mixing mask includes, for each pixel location, a different mixing ratio determined based on saliency values of the plurality of images at a same pixel location, wherein elements of the two-dimensional mixing mask indicate a saliency ratio between the plurality of images at the same pixel location;

generate, using the rearrangement offset and the two-dimensional mixing mask, a new mixed image by element-wise combining the plurality of images and a new mixed label from corresponding labels of the plurality of images; and

augment the dataset with the new mixed image and the new mixed label.

16 . The non-transitory computer-readable storage medium of claim 15 , wherein the computer-executable instructions, when executed by the at least one processor, further cause the device to:

provide each image of the plurality of images to a classifier model;

measure the full loss function between an output of the classifier model and the corresponding label of each image of the plurality of images; and

obtain the corresponding saliency map by calculating the gradient of the full loss function.

17 . The non-transitory computer-readable storage medium of claim 15 , wherein the computer-executable instructions, when executed by the at least one processor, further cause the device to:

identify the subset of arrangements from the plurality of possible arrangements, each arrangement of the subset of arrangements indicating a distinct positional translation of the plurality of images, the subset of arrangements comprising a predetermined portion of the plurality of possible arrangements;

calculate overall saliencies of sample images resulting from the distinct positional translations of the plurality of images indicated by the subset of arrangements; and

select, as the rearrangement offset, an arrangement from the subset of arrangements corresponding to a maximum overall saliency from among the overall saliencies.

18 . The method of claim 1 , wherein the computing of the two-dimensional mixing mask comprises:

computing, for each pixel location, the saliency ratio between the plurality of images by dividing a saliency value of a first image by a sum of saliency values of the plurality of images and a scalar parameter used to avoid a division-by-zero error.