Attribution and generation of saliency visualizations for machine-learning models
Methods, systems, devices, and tangible non-transitory computer readable media for saliency visualization are provided. The disclosed technology can include receiving a data input including a plurality of features. The data input can be segmented into regions. At least one of the regions can include two or more of the features. Attribution scores can be respectively generated for features of the data input. The attribution scores for each feature can be indicative of a respective saliency of such feature. A respective gain value for each region can be determined over one or more iterations based on the respective attribution scores associated with the features included in the region. Further, at each iteration one or more of the regions with the greatest gain values can be added to a saliency mask. Furthermore, at each iteration a saliency visualization can be produced based on the saliency mask.
1. A computer-implemented method, comprising:
receiving, by a computing system comprising one or more processors, an input comprising a plurality of features;
segmenting, by the computing system, the input into a plurality of regions, at least some regions of the plurality of regions overlapping with one or more other regions of the plurality of regions;
determining, by the computing system, attribution scores for each of the plurality of features;
determining, by the computing system, respective gain values for each respective region of the plurality of regions based on a sum of the attribution scores for features among the plurality of features within the respective region divided by an area associated with the respective region;
based on a first gain value for a first region among the plurality of regions being greater than a second gain value for a second region among the plurality of regions, selecting, by the computing system, the first region to add to a saliency mask; and
outputting, by the computing system, the saliency mask.
2. The computer-implemented method of claim 1 , wherein the respective gain values for each respective region of the plurality of regions are determined by the computing system based on the sum of the attribution scores for features among the plurality of features within the respective region divided by the area of the respective region not already included in the saliency mask.
3. The computer-implemented method of claim 1 , wherein the attribution scores are determined according to an integrated gradients attribution technique.
4. The computer-implemented method of claim 1 , wherein the total area of the plurality of regions is greater than the total area of the input.
5. The computer-implemented method of claim 1 , wherein the plurality of features are pixels.
6. The computer-implemented method of claim 1 , wherein the saliency mask corresponds to a predetermined portion of the input.
7. The computer-implemented method of claim 1 , further comprising applying the saliency mask to the input to classify an object associated with the first region.
8. The computer-implemented method of claim 1 , wherein, after adding the first region to the saliency mask, the method further comprises:
determining, by the computing system, respective updated gain values for each respective other region of the plurality of regions other than the first region, based on the attribution scores for features among the plurality of features within the respective other region; and
based on an updated second gain value for the second region among the plurality of regions being greater than an updated third gain value for a third region among the plurality of regions, selecting, by the computing system, the second region to add to the saliency mask to obtain an updated saliency mask.
9. The computer-implemented method of claim 8 , wherein the updated third gain value for the third region is based on a sum of the attribution scores for features among the plurality of features within the third region divided by an area of portions of the third region which are not added to the saliency mask.
10. The computer-implemented method of claim 8 , wherein the updated third gain value for the third region is based on a sum of the attribution scores for features among the plurality of features within the third region divided by a union of an area of the third region and the saliency mask.
11. The computer-implemented method of claim 8 , further comprising applying the updated saliency mask to the input to classify one or more objects associated with the first region and the second region.
12. The computer-implemented method of claim 1 , wherein the segmenting the input into the plurality of regions comprises:
performing, by the computing system, a plurality of iterations of a segmentation technique on the input respectively with a plurality of different segmentation parameter values.
13. The computer-implemented method of claim 1 , wherein:
the input includes an image and the plurality of features include a plurality of pixels from the image, and
an attribution score for a pixel among the plurality of pixels is based on a difference between a value of the pixel and a value of a first baseline.
14. The computer-implemented method of claim 13 , wherein an attribution score for a pixel among the plurality of pixels is further based on a difference between the value of the pixel and a value of a second baseline.
15. The computer-implemented method of claim 14 , wherein the first baseline is a black baseline and the second baseline is a white baseline.
16. The computer-implemented method of claim 1 , further comprising:
sequentially adding regions among the plurality of regions to the saliency mask according to gain values for each of the regions, wherein a region having a gain value greater than gain values of remaining regions among the plurality of regions is added to the saliency mask before the remaining regions.
17. The computer-implemented method of claim 16 , wherein regions among the plurality of regions are sequentially added to the saliency mask until a threshold number of regions are added.
18. The computer-implemented method of claim 16 , wherein regions among the plurality of regions are sequentially added to the saliency mask until the saliency mask contains a threshold area of the input.
19. A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions that when executed by the one or more processors cause the computing system to perform operations comprising:
receiving an input comprising a plurality of features;
segmenting the input into a plurality of regions, at least some regions of the plurality of regions overlapping with one or more other regions of the plurality of regions;
determining attribution scores for each of the plurality of features;
determining respective gain values for each respective region of the plurality of regions based on a sum of the attribution scores for features among the plurality of features within the respective region divided by an area associated with the respective region;
based on a first gain value for a first region among the plurality of regions being greater than a second gain value for a second region among the plurality of regions, selecting the first region to add to a saliency mask; and
outputting the saliency mask.
20. One or more non-transitory computer-readable media that store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising:
receiving an input comprising a plurality of features;
segmenting the input into a plurality of regions, at least some regions of the plurality of regions overlapping with one or more other regions of the plurality of regions;
determining attribution scores for each of the plurality of features;
determining respective gain values for each respective region of the plurality of regions based on a sum of the attribution scores for features among the plurality of features within the respective region divided by an area associated with the respective region;
based on a first gain value for a first region among the plurality of regions being greater than a second gain value for a second region among the plurality of regions, selecting the first region to add to a saliency mask; and
outputting the saliency mask.