Generating ground truths for generative AI applications
A first neural network is trained to generate a ground truth using a small set of example images that illustrate the goal ground truth output images, which can be full-body images of people in an AR style. The first neural network is used to generate ground truth output images from random input images. Example methods of the first neural network include determining poses in input images, changing values of pixels within areas of the input images, inputting the poses, the areas of the changed input images, and a text prompt describing the input images, into a neural network, to generate output images. The methods further include determining losses between the output images and the input images and updating weights of the neural network based on the losses. A second neural network is then trained using the generated ground truth. And, an application is generated that uses the second neural network.
1 . A computing device comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, configure the one or more processors to perform operations comprising:
determining a pose of a person depicted in an input image;
changing values of pixels within areas of the input image to generate an inpainted input image;
inputting the pose, the inpainted input image, and a text prompt describing the inpainted input image, into a neural network, to generate an output image;
determining losses between the output image and the input image; and
updating weights of the neural network based on the losses.
2 . The computing device of claim 1 , wherein the changing the values of pixels comprises:
changing the values of pixels to indicate a same value.
3 . The computing device of claim 1 , wherein the determining the pose for the input image comprises:
identifying key points of a person depicted in the input image; and
connecting the key points.
4 . The computing device of claim 1 , wherein the text prompt describing the input image describes one or more characteristics of the input image.
5 . The computing device of claim 1 , wherein the operations further comprise:
determining one or more keywords for the input image; and
wherein the inputting further comprising: inputting the one or more keywords.
6 . The computing device of claim 1 , wherein the operations further comprise:
determining keywords for the input image, wherein the keywords describe a characteristic of a person depicted within the input image, and wherein the input to the neural network further comprises the keywords.
7 . The computing device of claim 6 , wherein the keywords comprise at least one of a gender of the person, a skin tone of the person, a hair color of the person, an indication if gloves are worn by the person, an indication if glasses are worn by the person, and an indication of a color of shoes worn by the person.
8 . The computing device of claim 1 , wherein the operations further comprise:
determining body part masks for a plurality of ground truth input images;
determining bounding boxes for the plurality of ground truth input images, each bounding box comprising a person depicted within a corresponding ground truth input image of the plurality of ground truth input images;
determining areas for the plurality of ground truth input images, the areas comprising a difference between the bounding boxes and corresponding body part masks;
changing values of pixels within the areas of the plurality of ground truth input images to generate a plurality of inpainted input images;
determining poses for the plurality of ground truth input images; and
inputting the poses, the plurality of inpainted input images, and the text prompt, into the neural network with the updated weights, to generate a plurality of ground truth output images.
9 . The computing device of claim 8 , wherein the values of pixels are changed to a same value as the values of pixels within the areas of the plurality of input images.
10 . The computing device of claim 8 , wherein the operations further comprise:
blending the body part masks with the plurality of ground truth output images.
11 . The computing device of claim 10 , wherein the operations further comprise:
extracting body parts from the plurality of ground truth input images; and
blending the extracted body parts with corresponding ground truths of the plurality of ground truth output images.
12 . The computing device of claim 11 , wherein the blending is outside the areas.
13 . The computing device of claim 8 , wherein the operations further comprise:
determining third poses for the plurality of ground truth output images;
comparing the second poses with corresponding third poses; and
removing a ground truth output image of the plurality of ground truth output images in response to differences between a second pose of a ground truth input image and a third pose of the third poses of a corresponding ground truth output image transgressing a threshold.
14 . The computing device of claim 8 , wherein the operations further comprise:
determining keywords for the plurality of ground truth input images, wherein the keywords describe a characteristic of a person depicted within a corresponding ground truth input image of the plurality of ground truth input images, and wherein the input to the neural network further comprises the keywords.
15 . The computing device of claim 8 , wherein the neural network is a first neural network, and wherein the operations further comprise:
inputting the plurality of ground truth input images to generate a plurality of second output images, into a second neural network, to generate a plurality of second output images;
determining losses between the plurality of second output images and corresponding ground truth output images of the plurality of ground truth output images; and
updating weights of the second neural network based on the losses.
16 . A non-transitory computer-readable storage medium including instructions that, when processed by one or more processors, configure the one or more processors to perform operations comprising:
determining a pose of a person depicted in an input image;
changing values of pixels within areas of the input image to generate an inpainted input image;
inputting the pose, the inpainted input image, and a text prompt describing the input image, into a neural network, to generate an output image;
determining losses between the output image and the input image; and
updating weights of the neural network based on the losses.
17 . The non-transitory computer-readable storage medium of claim 16 , wherein the changing the values of the pixels comprises:
changing the values of the pixels within the areas to indicate a same value.
18 . The non-transitory computer-readable storage medium of claim 16 , wherein the determining the pose for the input image comprises:
identifying key points of a person depicted in the input image; and
connecting the key points.
19 . A method comprising:
determining a pose of a person depicted in an input image;
changing values of pixels within areas of the input image to generate an inpainted input image;
inputting the pose, the inpainted input image, and a text prompt describing the inpainted input image, into a neural network, to generate an output image;
determining losses between the output image and the input image; and
updating weights of the neural network based on the losses.
20 . The apparatus of claim 1 , wherein the text prompt is a first text prompt, and the output image is a first output image, and wherein the operations further comprise:
inputting a second text prompt into the neural network to generate a second output image.