Controllable three-dimensional object generation
Implementations disclosed herein are directed to using a machine learning model to generate three-dimensional models of objects. In some implementations, a computer implemented method can include the steps of: generating, by a machine-learning model, a three-dimensional model of an object from a three-dimensional model of an initial object and conditional input(s) specifying target properties of the generated object; generating two-dimensional image(s) of the generated object from the three-dimensional model of the generated object; generating a respective image embedding for each of the two-dimensional image(s) of the generated object using an image embedding model; generating a respective conditional embedding for each of the conditional input(s); and updating parameters of the machine-learning model based on a comparison of the respective conditional embedding for each of the conditional input(s) and the respective image embeddings for each of the two-dimensional image(s) of the generated object.
1 . A computer implemented method comprising:
for a plurality of iterations:
generating, by a machine-learning model, a three-dimensional model of an object from a three-dimensional model of an initial object and one or more conditional inputs specifying target properties of the generated three-dimensional model of the object, wherein the one or more conditional inputs comprises an image and/or a text description, wherein the three-dimensional model of the initial object and the one or more conditional inputs are selected from a set of training data comprising a plurality of training examples, each training example comprising a respective initial three-dimensional model of an object and a respective one or more sets of conditioning data;
generating one or more two-dimensional images of the generated object from the three-dimensional model of the generated object using a differentiable renderer;
generating a respective image embedding for each of the one or more two-dimensional images of the generated object using an image embedding model;
generating a respective conditional embedding for each of the one or more conditional inputs using one or more embedding models, wherein the respective conditional embeddings for each of the one or more conditional inputs and respective image embeddings for each of the one or more two-dimensional images of the generated object occupy a shared latent space; and
updating parameters of the machine-learning model based on a comparison of the respective conditional embedding for each of the one or more conditional inputs and the respective image embeddings for each of the one or more two-dimensional images of the generated object.
2 . The method of claim 1 , wherein updating parameters of the machine-learning model based on a comparison of the respective conditional embedding for each of the one or more conditional inputs and the respective image embeddings for each of the one or more two-dimensional images of the generated object comprises:
determining a loss between the conditional embeddings and the image embeddings for each of the one or more two-dimensional images of the generated object using an objective function; and
determining parameter updates to the parameters of the machine-learning model based on the objective function,
wherein the objective function comprises one or more sematic similarity losses.
3 . The method of claim 2 , wherein:
the conditioning input comprises an image and a text description; and
the objective function comprises:
a text semantic similarity loss; and
an image sematic similarity loss.
4 . The method of claim 3 , wherein the objective function further comprises a style loss between a conditioning image and the one or more two-dimensional images of the generated object.
5 . The method of claim 4 , wherein the objective function comprises a weighted sum of the one or more sematic similarity losses and the style loss.
6 . The method of claim 1 , further comprising iterating the method until a threshold condition is satisfied.
7 . The method of claim 1 , wherein:
the one or more conditional inputs comprises a text input describing an edit to the three-dimensional model of the initial object; and
the three-dimensional model of the object comprises an edited version of the three-dimensional model of the initial object with the described edit.
8 . The method of claim 1 , wherein:
the initial three-dimensional model is an environment;
the one or more conditional inputs comprises a text input describing an environment layout;
the one or more conditional inputs further comprises an asset library comprising three-dimensional models of potential objects in the environment; and
the three-dimensional model of the object comprises a further environment with the environment layout described in the text input.
9 . The method of claim 1 , wherein:
the initial three-dimensional model comprises a plurality of three-dimensional human models;
the one or more conditional inputs comprises a text input describing an action;
the three-dimensional model of the object comprises a three-dimensional animation of a human performing the action; and
generating one or more two-dimensional images of the generated object from the three-dimensional model of the generated object using the differentiable renderer comprises generating one or more two-dimensional animations of the human performing the action.
10 . Apparatus comprising:
one or more processors;
and a memory, the memory storing computer readable instructions that, when executed by the on more processors causes the apparatus to perform a method comprising:
for a plurality of iterations:
generating, by a machine-learning model, a three-dimensional model of an object from a three-dimensional model of an initial object and one or more conditional inputs specifying target properties of the generated three-dimensional model of the object, wherein the one or more conditional inputs comprises an image and/or a text description, wherein the three-dimensional model of the initial object and the one or more conditional inputs are selected from a set of training data comprising a plurality of training examples, each training example comprising a respective initial three-dimensional model of an object and a respective one or more sets of conditioning data;
generating one or more two-dimensional images of the generated object from the three-dimensional model of the generated object using a differentiable renderer;
generating a respective image embedding for each of the one or more two-dimensional images of the generated object using an image embedding model;
generating a respective conditional embedding for each of the one or more conditional inputs using one or more embedding models, wherein the respective conditional embeddings for each of the one or more conditional inputs and respective image embeddings for each of the one or more two-dimensional images of the generated object occupy a shared latent space; and
updating parameters of the machine-learning model based on a comparison of the respective conditional embedding for each of the one or more conditional inputs and the respective image embeddings for each of the one or more two-dimensional images of the generated object.
11 . The apparatus of claim 10 , wherein updating parameters of the machine-learning model based on a comparison of the respective conditional embedding for each of the one or more conditional inputs and the respective image embeddings for each of the one or more two-dimensional images of the generated object comprises:
determining a loss between the conditional embeddings and the image embeddings for each of the one or more two-dimensional images of the generated object using an objective function; and
determining parameter updates to the parameters of the machine-learning model based on the objective function,
wherein the objective function comprises one or more sematic similarity losses.
12 . The apparatus of claim 11 wherein:
the conditioning input comprises an image and a text description; and
the objective function comprises:
a text semantic similarity loss; and
an image sematic similarity loss.
13 . The apparatus of claim 12 , wherein the objective function further comprises a style loss between a conditioning image and the one or more two-dimensional images of the generated object.
14 . The apparatus of claim 13 , wherein the objective function comprises a weighted sum of the one or more sematic similarity losses and the style loss.
15 . The apparatus of claim 10 , further comprising iterating the method until a threshold condition is satisfied.
16 . The apparatus of claim 10 , wherein:
the one or more conditional inputs comprises a text input describing an edit to the three-dimensional model of the initial object; and
the three-dimensional model of the object comprises an edited version of the three-dimensional model of the initial object with the described edit.
17 . The apparatus of claim 10 , wherein:
the initial three-dimensional model is an environment;
the one or more conditional inputs comprises a text input describing an environment layout;
the one or more conditional inputs further comprises an asset library comprising three-dimensional models of potential objects in the environment; and
the three-dimensional model of the object comprises a further environment with the environment layout described in the text input.
18 . The apparatus of claim 10 , wherein:
the initial three-dimensional model comprises a plurality of three-dimensional human models;
the one or more conditional inputs comprises a text input describing an action;
the three-dimensional model of the object comprises a three-dimensional animation of a human performing the action; and
generating one or more two-dimensional images of the generated object from the three-dimensional model of the generated object using the differentiable renderer comprises generating one or more two-dimensional animations of the human performing the action.