Dynamic model adaptation customized for individual users
Described is a system for dynamically applying model adaptations customized for individual users by detecting an image of a first real-world object from a camera feed, detecting landmarks on the first real-world object, and processing the landmarks on the first real-world object using a generative machine learning model to generate a first custom image template for the first real-world object where portions of the first custom image template are populated with visual content placed based on the first custom image template. The system then applies a content augmentation based on the first custom image template to the camera feed.
1 . A system comprising:
at least one processor; and
at least one memory component storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
identifying a prompt of a first user indicating an intent for a first content augmentation;
detecting an image of a first real-world object from a camera feed of a user system associated with the first user;
detecting one or more landmarks on the first real-world object;
processing data associated with the one or more landmarks on the first real-world object and the prompt using a generative machine learning model to generate a first custom image template for the first real-world object in which one or more portions of the first custom image template are populated with visual content placed based on the first custom image template, the first custom image template being populated based on an artificial texture generated by the generative machine learning model using the prompt; and
applying the first content augmentation on at least a portion of the first real-world object based on the first custom image template to the camera feed.
2 . The system of claim 1 , wherein the operations further comprise continuously applying the first content augmentation on the first real-world object in response to continuous capture of the camera feed.
3 . The system of claim 1 , wherein the first real-world object is the first user, and the one or more landmarks include facial features on a face of the first user.
4 . The system of claim 1 , wherein the operations further comprise:
detecting a second real-world object from the camera feed;
generating a second custom image template by processing data associated with one or more landmarks on the second real-world object; and
applying a second content augmentation on at least a portion of the second real-world object based on the second custom image template to the camera feed.
5 . The system of claim 4 , wherein the first real-world object is the first user, and the second real-world object is a second user.
6 . The system of claim 1 , wherein the operations further comprise estimating depth data for the one or more landmarks, wherein the depth data represents a distance of the one or more landmarks to the user system, wherein the data processed using the generative machine learning model further includes the estimated depth data.
7 . The system of claim 1 , wherein identifying the prompt comprises receiving a voice or text command from the first user.
8 . The system of claim 1 , wherein identifying the prompt comprises determining an intent of the first user based on multimodal memory embeddings associated with the first user.
9 . The system of claim 1 , wherein the operations further comprise:
identifying an updated prompt of the first user indicating an updated intent for a second content augmentation;
processing data associated with the one or more landmarks on the first real-world object using the generative machine learning model to generate a second custom image template for the first real-world object in which one or more portions of the second custom image template are populated with visual content placed based on the second custom image template; and
replacing the first content augmentation with a second content augmentation to apply on at least a portion of the first real-world object based on the second custom image template to the camera feed.
10 . The system of claim 9 , wherein the prompt comprises a first collection of words that include a first adjective, and the updated prompt comprises a second collection of words that include a second adjective, wherein the visual content populated on the second custom image template corresponds to the second adjective.
11 . The system of claim 1 , wherein the generative machine learning model includes a stable diffusion model.
12 . The system of claim 1 , wherein the generative machine learning model is trained to generate custom image templates specific to a particular individual's face based on facial landmarks identified in images, wherein content augmentations are applied to user faces based on the generated custom image templates.
13 . The system of claim 1 , wherein the operations further comprise training the generative machine learning model by:
collecting a dataset of images related to real-world objects;
collecting landmark data for each real-world object in the dataset of real-world objects;
inputting the dataset of images and landmark data to the generative machine learning model;
using a denoising score matching objective to determine a loss function; and
updating one or more parameters in the generative machine learning model to reduce the loss function.
14 . The system of claim 1 , wherein the operations further comprise generating the first content augmentation by overlaying the first custom image template on at least a portion of the first real-world object.
15 . The system of claim 14 , wherein generating the first content augmentation comprises adjusting a 3D mesh for the first real-world object based on the first custom image template.
16 . The system of claim 15 , wherein adjusting the 3D mesh comprises generating a point cloud of 3D points representing spatial information by mapping pixel coordinates of the first custom image template with the pixel coordinates of depth information in a second custom image template.
17 . The system of claim 1 , wherein the generative machine learning model is trained to generate custom image templates to correspond with one or more adjectives identified in prompts.
18 . A method comprising:
training a generative machine learning model by:
collecting a dataset of images related to real-world objects;
collecting landmark data for each real-world object in the dataset of real-world objects;
inputting the dataset of images and landmark data to the generative machine learning model;
using a denoising score matching objective to determine a loss function; and
updating one or more parameters in the generative machine learning model to reduce the loss function;
detecting an image of a first real-world object from a camera feed of a user system associated with a first user;
detecting one or more landmarks on the first real-world object;
processing data associated with the one or more landmarks on the first real-world object using the generative machine learning model to generate a first custom image template for the first real-world object in which one or more portions of the first custom image template are populated with visual content placed based on the first custom image template; and
applying a first content augmentation on at least a portion of the first real-world object based on the first custom image template to the camera feed.
19 . A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
detecting an image of a first real-world object from a camera feed of a user system associated with a first user;
detecting one or more landmarks on the first real-world object;
processing data associated with the one or more landmarks on the first real-world object using a generative machine learning model to generate a first custom image template for the first real-world object in which one or more portions of the first custom image template are populated with visual content placed based on the first custom image template, wherein the generative machine learning model is trained to generate custom image templates to correspond with one or more adjectives identified in prompts; and
applying a first content augmentation on at least a portion of the first real-world object based on the first custom image template to the camera feed.