Systems and methods for generating text descriptive of digital images
In implementation of techniques for generating text descriptive of digital images, a computing device implements a captioning system to generate a text description of an image using a machine learning model. The captioning system identifies a negative word in the text description of the image based on a quantitative risk level for the negative word. Based on the text description of the image and the quantitative risk level, the captioning system determines an amount of suppression for use of the negative word. The captioning system then generates an updated text description of the image by applying the amount of suppression to the machine learning model to change a likelihood of using the negative word.
1 . A method comprising:
generating, by a processing device, a text description of an image using a machine learning model;
identifying, by the processing device, a negative word describing an entity in the text description of the image and a category corresponding to the negative word;
determining, by the processing device, a quantitative risk level for the negative word based on use of the negative word in the text description;
determining, by the processing device, a threshold risk level corresponding to the category from multiple different threshold risk levels;
determining, by the processing device, an amount of suppression for use of the negative word based on the text description of the image and the quantitative risk level meeting the threshold risk level; and
generating, by the processing device, an updated text description of the image including replacing the negative word with an alternative word by applying the amount of suppression to the machine learning model.
2 . The method of claim 1 , wherein the machine learning model is biased based on the amount of suppression.
3 . The method of claim 1 , wherein the text description of the image is formatted for audible output.
4 . The method of claim 1 , wherein the text description of the image is used to generate a database of images.
5 . The method of claim 1 , further comprising generating a caption using the text description of the image for display associated with the image.
6 . The method of claim 1 , wherein the negative word is identified from a list of negative words.
7 . The method of claim 6 , wherein the list of negative words is organized by the quantitative risk level.
8 . The method of claim 1 , wherein the negative word is a geographic location and the text description of the image is adjusted to replace the geographic location with a generalized geographic location.
9 . The method of claim 1 , wherein the amount of suppression is based on a confidence of use of the negative word in the text description.
10 . The method of claim 1 , wherein the threshold risk level depends on a type of the negative word.
11 . The method of claim 1 , wherein the machine learning model is biased based on the amount of suppression by multiplying post-softmax likelihoods or pre-softmax logits by a factor inversely proportional to the quantitative risk level for the negative word.
12 . The method of claim 1 , further comprising training the machine learning model to reduce selection of the negative word.
13 . A system comprising:
a memory component; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
generating a text description of an image using a machine learning model;
identifying a negative word describing an entity in the text description of the image and a category corresponding to the negative word;
determining a quantitative risk level for use of the negative word based on use of the negative word in the text description;
determining a threshold risk level corresponding to the category from multiple different threshold risk levels;
determining an amount of suppression for use of the negative word based on the text description of the image and the quantitative risk level meeting the threshold risk level; and
generating an updated text description of the image by replacing the negative word with an alternative word by applying the amount of suppression to the machine learning model.
14 . The system of claim 13 , further comprising mapping the quantitative risk level to the amount of suppression.
15 . The system of claim 13 , wherein the text description of the image is formatted for audible output.
16 . The system of claim 13 , wherein the text description of the image is used to generate a database of images.
17 . The system of claim 13 , further comprising generating a caption using the text description of the image for display associated with the image.
18 . A non-transitory computer-readable storage medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
generating a text description of an image using a machine learning model;
identifying a negative word describing an entity in the text description of the image and a category corresponding to the negative word;
determining a quantitative risk level for use of the negative word based on use of the negative word in the text description;
determining a threshold risk level corresponding to the category from multiple different threshold risk levels;
determining an amount of suppression for use of the negative word based on the text description of the image and the quantitative risk level meeting the threshold risk level; and
generating an updated text description of the image including replacing the negative word with an alternative word by applying the amount of suppression to the machine learning model.
19 . The non-transitory computer-readable storage medium of claim 18 , wherein the text description of the image is used to generate a database of images.
20 . The non-transitory computer-readable storage medium of claim 18 , wherein the negative word is identified from a list of negative words that is organized by the quantitative risk level.