Output accuracy refinement mechanism for visual prompting

1 . A method comprising:

inputting a raw image into a machine learning model;

receiving a predicted image comprising a plurality of masks as output from the machine learning model, each mask of the plurality of masks surrounding a plurality of pixels predicted to correspond to a given classification predicted by the machine learning model;

for each mask of the plurality of masks:

determining an extent of the mask using a connected components analysis;

allocating a number of points based on a relative extent of the mask as compared to given extents of each other mask of the plurality of masks;

selecting representative points corresponding to the number of points that are representative of an area within the mask;

inputting the raw image and the representative points from the predicted image into a segmentation model and receiving output from the segmentation model including, for each mask, a segment around the representative points;

inputting output from the segmentation model and class predictions from the predicted image into a fusion model;

receiving, as output from the fusion model, revised boundaries for each mask of the plurality of masks, the revised boundaries generated by fusing each mask with a boundary determined from the output from the segmentation model; and

outputting classifications for each pixel of the raw image based on the revised boundaries.

2 . The method of claim 1 , wherein generating the segment around the representative points comprises drawing a boundary corresponding to a shape within the raw image.

3 . The method of claim 2 , wherein the fusion model further generates the classifications by:

for each segment, determining a representative class that corresponds to a majority of pixels within the segment; and

assigning the representative class to the segment.

4 . The method of claim 3 , wherein each mask of the plurality of masks is ranked from largest to smallest, and wherein the representative class is assigned in decreasing order from a largest segment to a smallest segment.

5 . The method of claim 1 , wherein selecting the representative points comprises determining a core set for pixels within the mask, the core set determined by:

obtaining a subset of feature vectors for the pixels within the mask by sampling a plurality of feature vectors for each of the pixels within the mask; and

adding additional ones of the plurality of feature vectors to the subset until a criterion is reached.

6 . The method of claim 1 , wherein selecting the representative points comprises applying a distance transformation for the pixels within the mask.

7 . A non-transitory computer-readable medium comprising memory with instructions encoded thereon that, when executed, cause one or more processors to perform operations, the instructions comprising instructions to:

input a raw image into a machine learning model;

receive a predicted image comprising a plurality of masks as output from the machine learning model, each mask of the plurality of masks surrounding a plurality of pixels predicted to correspond to a given classification predicted by the machine learning model;

for each mask of the plurality of masks:

determine an extent of the mask using a connected components analysis;

allocate a number of points based on a relative extent of the mask as compared to given extents of each other mask of the plurality of masks;

select representative points corresponding to the number of points that are representative of an area within the mask;

input the raw image and the representative points from the predicted image into a segmentation model and receiving output from the segmentation model including, for each mask, a segment around the representative points;

input output from the segmentation model and class predictions from the predicted image into a fusion model;

receive, as output from the fusion model, revised boundaries for each mask of the plurality of masks, the revised boundaries generated by fusing each mask with a boundary determined from the output from the segmentation model; and

output classifications for each pixel of the raw image based on the revised boundaries.

8 . The non-transitory computer-readable medium of claim 7 , wherein generating the segment around the representative points comprises drawing a boundary corresponding to a shape within the raw image.

9 . The non-transitory computer-readable medium of claim 8 , wherein the fusion model further generates the classifications by:

for each segment, determining a representative class that corresponds to a majority of pixels within the segment; and

assigning the representative class to the segment.

10 . The non-transitory computer-readable medium of claim 9 , wherein each mask of the plurality of masks is ranked from largest to smallest, and wherein the representative class is assigned in decreasing order from a largest segment to a smallest segment.

11 . The non-transitory computer-readable medium of claim 7 , wherein selecting the representative points comprises determining a core set for pixels within the mask, the core set determined by:

obtaining a subset of feature vectors for the pixels within the mask by sampling a plurality of feature vectors for each of the pixels within the mask; and

adding additional ones of the plurality of feature vectors to the subset until a criterion is reached.

12 . The non-transitory computer-readable medium of claim 7 , wherein selecting the representative points comprises applying a distance transformation for the pixels within the mask.

13 . A system comprising:

memory with instructions encoded thereon; and

one or more processors that, when executing the instructions, are caused to perform operations comprising:

inputting a raw image into a machine learning model;

receiving a predicted image comprising a plurality of masks as output from the machine learning model, each mask of the plurality of masks surrounding a plurality of pixels predicted to correspond to a given classification predicted by the machine learning model;

for each mask of the plurality of masks:

determining an extent of the mask using a connected components analysis;

allocating a number of points based on a relative extent of the mask as compared to given extents of each other mask of the plurality of masks;

selecting representative points corresponding to the number of points that are representative of an area within the mask;

inputting the raw image and the representative points from the predicted image into a segmentation model and receiving output from the segmentation model including, for each mask, a segment around the representative points;

inputting output from the segmentation model and class predictions from the predicted image into a fusion model;

receiving, as output from the fusion model, revised boundaries for each mask of the plurality of masks, the revised boundaries generated by fusing each mask with a boundary determined from the output from the segmentation model; and

outputting classifications for each pixel of the raw image based on the revised boundaries.

14 . The system of claim 13 , wherein generating the segment around the representative points comprises drawing a boundary corresponding to a shape within the raw image.

15 . The system of claim 14 , wherein the fusion model further generates the classifications by:

for each segment, determining a representative class that corresponds to a majority of pixels within the segment; and

assigning the representative class to the segment.

16 . The system of claim 15 , wherein each mask of the plurality of masks is ranked from largest to smallest, and wherein the representative class is assigned in decreasing order from a largest segment to a smallest segment.

17 . The system of claim 13 , wherein selecting the representative points comprises determining a core set for pixels within the mask, the core set determined by:

obtaining a subset of feature vectors for the pixels within the mask by sampling a plurality of feature vectors for each of the pixels within the mask; and

adding additional ones of the plurality of feature vectors to the subset until a criterion is reached.