Data attribution for diffusion models
Embodiments of the present disclosure perform training attribution by identifying a synthesized image and a training image, where the synthesized image was generated by an image generation model that was trained with the training image. A machine learning model computes first attribution features for the synthesized image using a first mapping layer and second attribution features for the training image using a second mapping layer that is different from the first mapping layer. Then, an attribution score is generated based on the first attribution features and the second attribution features, where the attribution score indicates a degree of influence for the training image on generating the synthesized image.
1 . A method comprising:
identifying a synthesized image and a training image, wherein the synthesized image was generated by an image generation model that was trained with the training image;
computing first attribution features based on the synthesized image using a first encoder and second attribution features based on the training image using a second encoder, wherein the first encoder includes a base encoder and a first mapping layer, and wherein the second encoder includes the base encoder and a second mapping layer that is different from the first mapping layer; and
generating an attribution score by computing a vector similarity between the first attribution features and the second attribution features, wherein the attribution score indicates a degree of influence for the training image on generating the synthesized image.
2 . The method of claim 1 , wherein computing the first attribution features and the second attribution features comprises:
generating first image features for the synthesized image and second image features for the training image using the base encoder of a machine learning model, wherein the first attribution features and the second attribution features are based on the first image features and the second image features, respectively.
3 . The method of claim 2 , wherein:
the first image features and the second image features comprise object features, style features, or a combination thereof.
4 . The method of claim 1 , further comprising:
receiving a query indicating the synthesized image; and
retrieving the training image in response to the query based on the attribution score.
5 . The method of claim 1 , further comprising:
computing a plurality of attribution scores for a plurality of training images, respectively; and
selecting the training image from the plurality of training images based on the plurality of attribution scores.
6 . The method of claim 1 , wherein:
the first mapping layer and the second mapping layer represent different affine transformations.
7 . The method of claim 1 , wherein:
the first mapping layer and the second mapping layer are trained using contrastive learning based on a positive sample pair including a first image used to train the image generation model and a second image generated by the image generation model.
8 . A method of training a machine learning model, the method comprising:
obtaining training data including an exemplar image used to train an image generation model and a synthesized image generated by the image generation model; and
training, using the training data, a machine learning model to generate an attribution score for synthesized images by training a first mapping layer of a first encoder of the machine learning model to generate first attribution features for the synthesized images and training a second mapping layer of a second encoder of the machine learning model to generate second attribution features for candidate training images used to train the image generation model, wherein the first encoder includes a base encoder and the first mapping layer, and wherein the second encoder includes the base encoder and the second mapping layer.
9 . The method of claim 8 , wherein:
the first mapping layer and the second mapping layer perform different affine transformations from each other.
10 . The method of claim 8 , wherein:
the training data includes a plurality of image pairs from a plurality of different image generation models.
11 . The method of claim 8 , wherein obtaining the training data comprises:
fine-tuning the image generation model based on the exemplar image; and
generating the synthesized image using the fine-tuned image generation model.
12 . The method of claim 8 , wherein training the machine learning model comprises:
computing a normalized temperature cross-entropy loss; and
updating parameters of the machine learning model based on the normalized temperature cross-entropy loss.
13 . The method of claim 8 , wherein:
the training comprises contrastive learning based on the exemplar image and the synthesized image.
14 . An apparatus comprising:
at least one processor; and
at least one memory including instructions executable by the at least one processor to perform operations comprising:
identifying a synthesized image and a training image, wherein the synthesized image was generated by an image generation model that was trained with the training image;
computing first attribution features based on the synthesized image using a first encoder and second attribution features based on the training image using a second encoder, wherein the first encoder includes a base encoder and a first mapping layer, and wherein the second encoder includes the base encoder and a second mapping layer that is different from the first mapping layer; and
generating an attribution score by computing a vector similarity between the first attribution features and the second attribution features, wherein the attribution score indicates a degree of influence for the training image on generating the synthesized image.
15 . The apparatus of claim 14 , further comprising:
a training component configured to receive training data including an exemplar image used to train the image generation model and the synthesized image generated by the image generation model and configured to train, using the training data, the first mapping layer and the second mapping layer to generate the first attribution features and the second attribution features, respectively.
16 . The apparatus of claim 14 , wherein:
the base encoder generates first image features for the synthesized image and second image features for the training image, and wherein the first attribution features and the second attribution features are based on the first image features and the second image features, respectively.
17 . The apparatus of claim 16 , wherein:
the base encoder is fixed during training of the first mapping layer and the second mapping layer.
18 . The apparatus of claim 16 , wherein:
the base encoder comprises a part of the image generation model used to generate the synthesized image.
19 . The apparatus of claim 14 , wherein:
the first mapping layer and the second mapping layer comprise linear layers.