Modifying source data to generate hyperreal synthetic content
Modifying source data to generate hyperreal synthetic content is described. Source data representing images of a body part of a subject may be modified to obtain training data, such as by modifying the body part in the images and/or by enhancing the images to improve the quality thereof. One or more machine learning models may be trained using the training data to obtain one or more trained machine learning models, and the trained model(s) is used to generate output data representing a synthetic body part based at least in part on input data representing an image featuring the body part of the subject, such as an image featuring the modified body part and/or an enhanced image featuring the body part. The output data is then used to generate media data corresponding to media content featuring the synthetic body part.
1 . A method comprising:
modifying, by one or more processors, source data representing images of a face of a first person to obtain training data representing the images of the first person with a modified face that substantially matches a size and a shape of a face of a second person;
training, by the one or more processors, a first machine learning model against a second machine learning model using the training data to obtain a first trained machine learning model and a second trained machine learning model, the first trained machine learning model associated with the first person and the second trained machine learning model associated with the second person;
providing, by the one or more processors, input data representing an image featuring the modified face of the first person to the first trained machine learning model to determine a point in a latent space of the first trained machine learning model that corresponds to the image featuring the modified face of the first person;
generating, by the one or more processors, using the second trained machine learning model based at least in part on the point in the latent space of the first trained machine learning model, output data representing a synthetic face of the second person; and
causing, by the one or more processors, media content to be displayed on a display based at least in part on the output data, the media content having been generated by swapping the modified face of the first person featured in the image with the synthetic face of the second person.
2 . The method of claim 1 , wherein the modifying is based at least in part on one or more offsets between the face of the first person and the face of the second person.
3 . The method of claim 1 , wherein the modifying comprises at least one of:
adjusting a size or a shape of the face of the first person in the images;
adjusting a size, a shape, a position, or an orientation of a feature of the face of the first person in the images; or
adjusting a spacing between features of the face of the first person in the images.
4 . The method of claim 1 , further comprising:
generating, using a third trained machine learning model, a two-dimensional (2D) displacement mapping between the face of the first person and the face of the second person,
wherein the modifying is based at least in part on the 2D displacement mapping.
5 . The method of claim 1 , further comprising:
analyzing, by the one or more processors, the source data to identify missing data;
generating, by the one or more processors, using a diffusion model, synthetic data that corresponds to the missing data; and
adding, by the one or more processors, the synthetic data to the training data,
wherein the first machine learning model is trained against the second machine learning model using the training data that includes the synthetic data.
6 . A system comprising:
one or more processors; and
memory storing computer-executable instructions that, when executed by the one or more processors, cause performance of operations comprising:
modifying source data representing images of a face of a first person to obtain training data representing the images of the first person with a modified face that substantially matches a size and a shape of a face of a second person;
training a first machine learning model against a second machine learning model using the training data to obtain a first trained machine learning model and a second trained machine learning model, the first trained machine learning model associated with the first person and the second trained machine learning model associated with the second person;
providing input data representing an image featuring the modified face of the first person to the first trained machine learning model to determine a point in a latent space of the first trained machine learning model that corresponds to the image featuring the modified face of the first person;
generating, using the second trained machine learning model based at least in part on the point in the latent space of the first trained machine learning model, output data representing a synthetic face of the second person; and
causing media content to be displayed on a display based at least in part on the output data, the media content having been generated by swapping the modified face of the first person featured in the image with the synthetic face of the second person.
7 . The system of claim 6 , wherein the modifying is based at least in part on one or more offsets between the face of the first person and the face of the second person.
8 . The system of claim 6 , wherein the modifying comprises at least one of:
adjusting a size or a shape of the face of the first person in the images;
adjusting a size, a shape, a position, or an orientation of a feature of the face of the first person in the images; or
adjusting a spacing between features of the face of the first person in the images.
9 . The system of claim 6 , the operations further comprising:
generating, using a third trained machine learning model, a two-dimensional (2D) displacement mapping between the face of the first person and the face of the second person,
wherein the modifying is based at least in part on the 2D displacement mapping.
10 . The system of claim 6 , the operations further comprising:
analyzing the source data to identify missing data;
generating, using a diffusion model, synthetic data that corresponds to the missing data; and
adding the synthetic data to the training data,
wherein the first machine learning model is trained against the second machine learning model using the training data that includes the synthetic data.
11 . One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause performance of operations comprising:
modifying source data representing images of a face of a first person to obtain training data representing the images of the first person with a modified face that substantially matches a size and a shape of a face of a second person;
training a first machine learning model against a second machine learning model using the training data to obtain a first trained machine learning model and a second trained machine learning model, the first trained machine learning model associated with the first person and the second trained machine learning model associated with the second person;
providing input data representing an image featuring the modified face of the first person to the first trained machine learning model to determine a point in a latent space of the first trained machine learning model that corresponds to the image featuring the modified face of the first person;
generating, using the second trained machine learning model based at least in part on the point in the latent space of the first trained machine learning model, output data representing a synthetic face of the second person; and
causing media content to be displayed on a display based at least in part on the output data, the media content having been generated by swapping the modified face of the first person featured in the image with the synthetic face of the second person.
12 . The one or more non-transitory computer-readable media of claim 11 , wherein the modifying is based at least in part on one or more offsets between the face of the first person and the face of the second person.
13 . The one or more non-transitory computer-readable media of claim 11 , wherein the modifying comprises at least one of:
adjusting a size or a shape of the face of the first person in the images;
adjusting a size, a shape, a position, or an orientation of a feature of the face of the first person in the images; or
adjusting a spacing between features of the face of the first person in the images.
14 . The one or more non-transitory computer-readable media of claim 11 , the operations further comprising:
generating, using a third trained machine learning model, a two-dimensional (2D) displacement mapping between the face of the first person and the face of the second person,
wherein the modifying is based at least in part on the 2D displacement mapping.
15 . The one or more non-transitory computer-readable media of claim 11 , the operations further comprising:
analyzing the source data to identify missing data;
generating, using a diffusion model, synthetic data that corresponds to the missing data; and
adding the synthetic data to the training data,
wherein the first machine learning model is trained against the second machine learning model using the training data that includes the synthetic data.