System for automated generation of facial shapes for virtual character models
Systems and methods are provided for enhanced face shape generation for archetypes of virtual entities based on generative modeling techniques. An example method includes training models based on synthetically generated faces and information associated with an authoring system. The modeling system being trained to reconstruct face shapes for virtual entities based on a latent space embedding of a face identity associated with specific archetypes.
1 . A computer-implemented method comprising:
receiving a request to automatically generate a first synthetic identity associated with at least one archetype class and a first virtual face model associated with the first synthetic identity;
accessing an archetype cluster model trained based on a plurality of human faces associated with individual archetypes, each human face being defined based on at least one archetype associated with a plurality of facial features, the archetype cluster model trained to identify an archetype cluster of a plurality of archetype clusters within latent space, wherein each archetype cluster is associated with defined ranges of latent feature values within the latent space, and generate a latent feature representation of individual human faces within an identified cluster, wherein the latent feature representation is associated with an identity of the virtual human face;
identifying, using the archetype cluster model, at least one archetype cluster within latent space associated with the at least one archetype class;
generating, using the archetype cluster model, a latent feature representation of the first synthetic identity, wherein the latent feature representation includes latent feature values within the defined ranges of based at least in part on the at least one archetype cluster, wherein the latent feature representation is associated with the first synthetic identity of the first synthetic virtual face;
accessing a decoding engine, the decoding engine trained to reconstruct authoring parameters for an authoring engine based on a latent feature representation of a human face;
generating, using the decoding engine, authoring parameters based at least in part on the latent feature representation of the first synthetic identity; and
generating, using the authoring engine, the first virtual face model of a first virtual face based at least in part on the authoring parameters, wherein the virtual face model has the first synthetic identity, wherein the first virtual face model is a mesh model.
2 . The computer-implemented method of claim 1 , wherein the latent feature representation is pseudo-randomly generated based on the latent space associated with the at least one archetype cluster of the at least one archetype class.
3 . The computer-implemented method of claim 2 , wherein the request further comprises requests to generate a plurality of virtual face models within at least one archetype class, and the latent feature representation of individual virtual faces is generated for each of the plurality of requested virtual face models.
4 . The computer-implemented method of claim 3 , wherein a virtual face identity corresponding to each of the plurality of virtual face models is pseudo-randomly generated, and each of the virtual face identities is generated from the latent space associated with the at least one cluster of the at least one archetype class, wherein each latent feature representation is separated from other latent feature representations by a defined threshold value.
5 . The computer-implemented method of claim 1 further comprising generating at least one facial characteristic associated with the mesh model of the first virtual face model, wherein the at least one facial characteristic is associated with the at least one archetype class.
6 . The computer-implemented method of claim 5 , wherein the at least one facial characteristic comprises at least one of skin texture, eye texture, hair mesh, or hair texture.
7 . The computer-implemented method of claim 1 , wherein the latent feature representation is a vector having a defined set of values.
8 . The computer-implemented method of claim 7 , wherein each archetype cluster is associated with at least one archetype, and, for each archetype cluster, each value of the set of values of the latent feature representation is associated with a range of values within the latent space.
9 . The computer-implemented method of claim 1 , wherein the first virtual face model is generated based on weights associated with a plurality of blendshapes that the define a shape of the mesh model.
10 . The computer-implemented method of claim 9 , wherein the authoring parameters define weights associated with the plurality of blendshapes.
11 . The computer-implemented method of claim 1 , wherein the archetype cluster model is a machine learning model generated using a deep neural network.
12 . Non-transitory computer storage media storing instructions that when executed by a system of one or more computers, cause the one or more computers to perform operations comprising:
receiving a request to automatically generate a first synthetic identity associated with at least one archetype class and a first virtual face model associated with the first synthetic identity;
accessing an archetype cluster model trained based on a plurality of human faces associated with individual archetypes, each human face being defined based on at least one archetype associated with a plurality of facial features, the archetype cluster model trained to identify an archetype cluster of a plurality of archetype clusters within latent space, wherein each archetype cluster is associated with defined ranges of latent feature values within the latent space, and generate a latent feature representation of individual human faces within an identified cluster, wherein the latent feature representation is associated with an identity of the virtual human face;
identifying, using the archetype cluster model, at least one archetype cluster within latent space associated with the at least one archetype class;
generating, using the archetype cluster model, a latent feature representation of the first synthetic identity, wherein the latent feature representation includes latent feature values within the defined ranges of the at least one archetype cluster, wherein the latent feature representation is associated with the first synthetic identity of the first synthetic virtual face;
accessing a decoding engine, the decoding engine trained to reconstruct authoring parameters for an authoring engine based on a latent feature representation of a human face;
generating, using the decoding engine, authoring parameters based at least in part on the latent feature representation of the first synthetic identity; and
generating, using the authoring engine, the first virtual face model of a first virtual face based at least in part on the authoring parameters, wherein the virtual face model has the first synthetic identity, wherein the virtual face model is a mesh model.
13 . The non-transitory computer storage media of claim 12 , wherein the latent feature representation is pseudo-randomly generated based on the latent space associated with the at least one archetype cluster of the at least one archetype class.
14 . The non-transitory computer storage media of claim 13 , wherein the request further comprises requests to generate a plurality of virtual face models within at least one archetype class, and the latent feature representation of individual virtual faces is generated for each of the plurality of requested virtual face models.
15 . The non-transitory computer storage media of claim 14 , wherein a virtual face identity corresponding to each of the plurality of virtual face models is generated from the latent space associated with the at least one cluster of the at least one archetype class, wherein each latent feature representation is separated from other latent feature representations by a defined threshold value.
16 . The non-transitory computer storage media of claim 12 further comprising generating at least one facial characteristic associated with the mesh model of the first virtual face model, wherein the at least one facial characteristic is associated with the at least one archetype class.
17 . The non-transitory computer storage media of claim 16 , wherein the at least one facial characteristic comprises at least one of skin texture, eye texture, hair mesh, or hair texture.
18 . The non-transitory computer storage media of claim 12 , wherein the latent feature representation is a vector having a defined set of values.
19 . The non-transitory computer storage media of claim 18 , wherein each archetype cluster is associated with at least one archetype, and, for each archetype cluster, each value of the set of values of the latent feature representation is associated with a range of values within the latent space.
20 . A system comprising one or more computers and non-transitory computer storage media storing instructions that when executed by the one or more computers, cause the one or more computers to perform operations comprising:
receiving a request to automatically generate a first synthetic identity associated with at least one archetype class and a first virtual face model associated with the first synthetic identity;
accessing an archetype cluster model trained based on a plurality of human faces associated with individual archetypes, each human face being defined based on at least one archetype associated with a plurality of facial features, the archetype cluster model trained to identify an archetype cluster of a plurality of archetype clusters within latent space, wherein each archetype cluster is associated with defined ranges of latent feature values within the latent space, and generate a latent feature representation of individual human faces within an identified cluster, wherein the latent feature representation is associated with an identity of the virtual human face;
identifying, using the archetype cluster model, at least one archetype cluster within latent space associated with the at least one archetype class;
generating, using the archetype cluster model, a latent feature representation of the first synthetic identity, wherein the latent feature representation includes latent feature values within the defined ranges of the at least one archetype cluster, wherein the latent feature representation is associated with the first synthetic identity of the first synthetic virtual face;
accessing a decoding engine, the decoding engine trained to reconstruct authoring parameters for an authoring engine based on a latent feature representation of a human face;
generating, using the decoding engine, authoring parameters based at least in part on the latent feature representation of the first synthetic identity; and
generating, using the authoring engine, the first virtual face model of a first virtual face based at least in part on the authoring parameters, wherein the virtual face model has the first synthetic identity, wherein the virtual face model is mesh model.