Weakly-supervised referring expression segmentation
The present disclosure relates to systems, methods, and non-transitory computer-readable media that modifies parameters of a fused feature extractor. In particular, the disclosed systems generate inferred masks from digital images and digital text prompts using a fused feature extractor. Furthermore, the disclosed systems identify a subset of the inferred masks that satisfy a validity threshold. Moreover, the disclosed systems generate an augmented training set by combining the subset of the inferred masks with a training set that includes the ground truth masks. Further, the disclosed systems generate object mask predictions from the augmented training set and determine ground truth and pseudo measures of loss by comparing the object mask predictions with the inferred masks and the ground truth masks. From the ground truth and pseudo measures of loss, the disclosed systems modify parameters of the fused feature extractors.
1 . A computer-implemented method comprising:
generating, utilizing a fused feature extractor, inferred masks from digital images and digital text prompts;
identifying a subset of the inferred masks satisfying a validity threshold;
generating an augmented training set by combining the subset of the inferred masks with a training set comprising ground truth masks;
generating, utilizing the fused feature extractor, object mask predictions from the augmented training set;
determining ground truth and pseudo measures of loss by comparing one or more object mask predictions with one or more inferred masks and one or more ground truth masks from the augmented training set; and
modifying parameters of the fused feature extractor based on the ground truth and pseudo measures of loss.
2 . The computer-implemented method of claim 1 , wherein generating the inferred masks from the digital images and the digital text prompts further comprises:
generating, utilizing an image encoder, an image encoding from a digital image and features extracted from a digital text prompt utilizing a text encoder; and
generating, utilizing the text encoder, a text encoding from the digital text prompt and features extracted from the digital image utilizing the image encoder.
3 . The computer-implemented method of claim 2 , further comprising:
generating a combined encoding from the image encoding and the text encoding; and
generating, utilizing a transformer model, an inferred mask that overlays the digital image based on the combined encoding.
4 . The computer-implemented method of claim 1 , wherein identifying the subset of the inferred masks satisfying the validity threshold further comprises:
detecting one or more objects within a digital image corresponding to an inferred mask; and
generating similarity scores between the one or more objects within the digital image and a digital text prompt corresponding to the inferred mask.
5 . The computer-implemented method of claim 4 , further comprising:
identifying, utilizing a spatial reasoning model, an object from the one or more objects within the digital image based on the similarity scores and a spatial reference in the digital text prompt corresponding to the inferred mask; and
comparing the identified object with the inferred mask to determine that the inferred mask matches a location of the identified object within the digital image.
6 . The computer-implemented method of claim 1 , wherein:
generating the augmented training set further comprises combining digital text prompts and digital images corresponding with the subset of the inferred masks satisfying the validity threshold with the training set; and
the training set further comprises digital images and digital text prompts corresponding with the ground truth masks.
7 . The computer-implemented method of claim 1 , wherein generating, utilizing the fused feature extractor, the object mask predictions further comprises:
generating, utilizing a cross-attention layer of an image encoder, a cross-attention map from a digital image and a digital text prompt of the augmented training set; and
comparing the cross-attention map with a ground truth mask or an inferred mask.
8 . The computer-implemented method of claim 7 , further comprising:
generating a cross-attention measure of loss between the cross-attention map and the ground truth mask to localize the cross-attention map within a specific object indicated by the digital text prompt; and
modifying parameters of the fused feature extractor and a transformer model based on the cross-attention measure of loss.
9 . The computer-implemented method of claim 7 , further comprising:
generating a cross-attention measure of loss between the cross-attention map and the ground truth mask or the inferred mask; and
generating a combined measure of loss by combining a cross-attention measure of loss with a cross-entropy loss to modify parameters of the fused feature extractor and a transformer model.
10 . The computer-implemented method of claim 1 , wherein the ground truth masks of the augmented training set corresponds to less than half of the augmented training set.
11 . The computer-implemented method of claim 1 , further comprising:
assigning a first weight to the subset of the inferred masks that is greater than a second weight assigned to the ground truth masks; and
modifying the parameters of the fused feature extractor based on the first weight and the second weight.
12 . A system comprising:
one or more memory devices; and
one or more processors coupled to the one or more memory devices, the one or more processors configured to cause the system to perform operations comprising:
generating, utilizing a fused feature extractor, inferred masks from digital images and digital text prompts;
identifying a subset of the inferred masks satisfying a validity threshold;
generating an augmented training set by combining the subset of the inferred masks with a training set comprising ground truth masks;
generating, utilizing the fused feature extractor, object mask predictions from the augmented training set;
determining ground truth and pseudo measures of loss by comparing one or more object mask predictions with one or more inferred masks and one or more ground truth masks from the augmented training set; and
modifying parameters of the fused feature extractor based on the ground truth and pseudo measures of loss.
13 . The system of claim 12 , wherein generating, utilizing the fused feature extractor, the object mask predictions further comprises:
generating, utilizing a cross-attention layer of an image encoder, a cross-attention map from a digital image and a digital text prompt of the augmented training set; and
comparing the cross-attention map with a ground truth mask or an inferred mask.
14 . The system of claim 13 , wherein the operations further comprise:
generating a cross-attention measure of loss between the cross-attention map and the ground truth mask to localize the cross-attention map within a specific object indicated by the digital text prompt; and
modifying parameters of the fused feature extractor and a transformer model based on the cross-attention measure of loss.
15 . The system of claim 14 , wherein generating the cross-attention measure of loss comprises generating a collapse-reduction term for retaining information from the cross-attention map within an object mask prediction.
16 . The system of claim 15 , wherein the operations further comprise:
generating a combined measure of loss by combining a cross-entropy measure of loss with the cross-attention measure of loss that comprises a localization term and the collapse-reduction term; and
modifying the parameters of the cross-attention layer and the transformer model with the combined measure of loss.
17 . A non-transitory computer-readable medium storing executable instructions which, when executed by at least one processing device, cause the at least one processing device to perform operations comprising:
generating, utilizing a fused feature extractor, inferred masks from digital images and digital text prompts;
identifying a subset of the inferred masks satisfying a validity threshold;
generating an augmented training set by combining the subset of the inferred masks with a training set comprising ground truth masks;
generating, utilizing the fused feature extractor, object mask predictions from the augmented training set;
determining ground truth and pseudo measures of loss by comparing one or more object mask predictions with one or more inferred masks and one or more ground truth masks from the augmented training set; and
modifying parameters of the fused feature extractor based on the ground truth and pseudo measures of loss.
18 . The non-transitory computer-readable medium of claim 17 , wherein generating the inferred masks from the digital images and the digital text prompts further comprises:
generating, utilizing an image encoder, an image encoding from a digital image and features extracted from a digital text prompt utilizing a text encoder; and
generating, utilizing the text encoder, a text encoding from the digital text prompt and features extracted from the digital image utilizing the image encoder.
19 . The non-transitory computer-readable medium of claim 18 , wherein the operations further comprise:
generating a combined encoding from the image encoding and the text encoding; and
generating, utilizing a transformer model, an inferred mask that overlays the digital image based on the combined encoding.
20 . The non-transitory computer-readable medium of claim 19 , wherein identifying the subset of the inferred masks satisfying the validity threshold further comprises:
detecting one or more objects within a digital image corresponding to an inferred mask; and
generating similarity scores between the one or more objects within the digital image and a digital text prompt corresponding to the inferred mask.