Automatic image quality evaluation
Examples disclosed herein describe techniques for automatic image quality evaluation. A first set of images generated by a first automated image generator and a second set of images generated by a second automated image generator are accessed. A first machine learning model generates a first quality indicator for each image in the first set of images and the second set of images. A second machine learning model generates a second quality indicator for each image in the first set of images and the second set of images. Based on the generated indicators, a first image from the first set of images and a second image from the second set of images are automatically selected and compared. A first ranking of the first automated image generator and the second automated image generator is generated based on the comparison, and ranking data is caused to be presented on a device.
1 . A system comprising:
a memory that stores instructions; and
one or more processors configured by the instructions to perform operations comprising:
accessing a first set of images generated by a first automated image generator and a second set of images generated by a second automated image generator, the first set of images and the second set of images having been automatically generated based on a first text prompt;
generating, by a first machine learning model that is trained to evaluate image quality using a first quality metric, a first quality indicator for each image in the first set of images and for each image in the second set of images;
generating, by a second machine learning model that is trained to evaluate image quality using a second quality metric, a second quality indicator for each image in the first set of images and for each image in the second set of images;
automatically selecting, based on the first quality indicators and the second quality indicators, a first image from the first set of images and a second image from the second set of images;
generating, based on an automatic comparison between the first image and the second image, a first ranking of the first automated image generator and the second automated image generator; and
causing presentation, on a device, of ranking data generated at least partially from the first ranking.
2 . The system of claim 1 , wherein the first automated image generator is a first text-to-image machine learning model, and the second automated image generator is a second text-to-image machine learning model.
3 . The system of claim 1 , the operations further comprising, prior to the accessing the first set of images:
generating, by the first automated image generator, the first set of images.
4 . The system of claim 3 , the operations further comprising, prior to the accessing the second set of images:
generating, by the second automated image generator, the second set of images.
5 . The system of claim 1 , wherein the first quality indicator and the second quality indicator are different, and the first quality indicator and the second quality indicator are each one of an aesthetic quality score, an alignment score, or a visual realism score.
6 . The system of claim 5 , the operations further comprising:
automatically determining, based on the respective first quality indicators and second quality indicators, a combined score for each image in the first set of images and a combined score for each image in the second set of images, wherein the first image has a highest combined score from among the first set of images and the second image has a highest combined score from among the second set of images.
7 . The system of claim 5 , wherein the aesthetic quality score for each image is generated by a Multi-Layer Perceptron (MLP) neural network.
8 . The system of claim 5 , wherein generation of the alignment score for each image comprises encoding the image and the first text prompt to obtain respective vectors, and automatically measuring a similarity between the respective vectors using a cosine similarity method.
9 . The system of claim 5 , wherein the visual realism score for each image is automatically generated using a Visual Question Answering (VQA) machine learning model.
10 . The system of claim 9 , wherein generating the visual realism score for each image further comprises automatically generating a prediction of whether the image includes one or more artifacts.
11 . The system of claim 1 , the operations further comprising:
generating, by a third machine learning model that is trained to evaluate image quality using a third quality metric, a third quality indicator for each image in the first set of images and for each image in the second set of images, wherein the first image and the second image are selected based on the first quality indicators, the second quality indicators, and the third quality indicators.
12 . The system of claim 11 , wherein the first quality indicator, the second quality indicator, and the third quality indicator are different, and the first quality indicator, the second quality indicator and the third quality indicator are each one of an aesthetic quality score, an alignment score, or a visual realism score.
13 . The system of claim 12 , the operations further comprising:
automatically determining, based on the respective first quality indicators, second quality indicators and third quality indicators, a combined score for each image in the first set of images and a combined score for each image in the second set of images, wherein the first image has a highest combined score from among the first set of images and the second image has a highest combined score from among the second set of images.
14 . The system of claim 1 , wherein the first ranking is based on an automatic comparison between the first quality indicator generated for the first image and the first quality indicator generated for the second image, and wherein the first quality metric is one of aesthetic quality, alignment, or visual realism.
15 . The system of claim 14 , the operations further comprising:
generating, based on an automatic comparison between the second quality indicator generated for the first image and the second quality indicator generated for the second image, a second ranking of the first automated image generator and the second automated image generator, the ranking data being generated at least partially from the first ranking and the second ranking, wherein the second quality metric differs from the first quality metric and is one of: aesthetic quality, alignment, or visual realism.
16 . The system of claim 3 , wherein the first automated image generator is a text-to-image machine learning model, and wherein the operations further comprise:
training the text-to-image machine learning model on a training data set comprising a plurality of training records, each training record comprising a training image and at least one corresponding text description for the training image.
17 . The system of claim 1 , the operations further comprising:
receiving, from a user device associated with a user, an image generation request comprising the first text prompt;
selecting, based on the automatic comparison between the first image and the second image, one of the first image or the second image as a selected image; and
causing presentation of the selected image on the user device.
18 . The system of claim 1 , the operations further comprising:
accessing a third set of images generated by the first automated image generator and a fourth set of images generated by the second automated image generator, the third set of images and the fourth set of images having been automatically generated based on a second text prompt;
generating, by the first machine learning model, a first quality indicator for each image in the third set of images and for each image in the fourth set of images;
generating, by the second machine learning model, a second quality indicator for each image in the third set of images and for each image in the fourth set of images;
automatically selecting, based on the first quality indicators and the second quality indicators for the third set of images and the fourth set of images, a third image from the third set of images and a fourth image from the fourth set of images; and
generating, based on an automatic comparison between the third image and the fourth image, a third ranking of the first automated image generator and the second automated image generator, wherein the ranking data is generated at least partially from the first ranking and the third ranking.
19 . A method comprising:
accessing, by one or more processors, a first set of images generated by a first automated image generator and a second set of images generated by a second automated image generator, the first set of images and the second set of images having been automatically generated based on a first text prompt;
generating, by the one or more processors and using a first machine learning model that is trained to evaluate image quality using a first quality metric, a first quality indicator for each image in the first set of images and for each image in the second set of images;
generating, by the one or more processors and using a second machine learning model that is trained to evaluate image quality using a second quality metric, a second quality indicator for each image in the first set of images and for each image in the second set of images;
automatically selecting, by the one or more processors, based on the first quality indicators and the second quality indicators, a first image from the first set of images and a second image from the second set of images;
generating, by the one or more processors, based on an automatic comparison between the first image and the second image, a first ranking of the first automated image generator and the second automated image generator; and
causing presentation, on a device, of ranking data generated at least partially from the first ranking.
20 . A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by at least one processor, cause the at least one processor to perform operations comprising:
accessing a first set of images generated by a first automated image generator and a second set of images generated by a second automated image generator, the first set of images and the second set of images having been automatically generated based on a first text prompt;
generating, by a first machine learning model that is trained to evaluate image quality using a first quality metric, a first quality indicator for each image in the first set of images and for each image in the second set of images;
generating, by a second machine learning model that is trained to evaluate image quality using a second quality metric, a second quality indicator for each image in the first set of images and for each image in the second set of images;
automatically selecting, based on the first quality indicators and the second quality indicators, a first image from the first set of images and a second image from the second set of images;
generating, based on an automatic comparison between the first image and the second image, a first ranking of the first automated image generator and the second automated image generator; and
causing presentation, on a device, of ranking data generated at least partially from the first ranking.