3D digital virtual character generation with facial feature preservation
Systems and methods for generating virtual characters for video conferencing are provided. In an example, a computing device accesses a source human face model, a target human face model, and a source virtual character face model as well as feature curves marked on the source human face model and the source virtual character face model. The computing device deforms the source virtual character face model based on the source human face model and the target human face model to generate a target virtual character face model. Deforming the source virtual character face model includes minimizing a loss function that includes terms defined based on the feature curves to preserve features of the source virtual character face model on the target virtual character face model. The computing device further renders the target virtual character face model.
1 . A computer implemented method for generating a virtual character, the method comprising:
accessing a source human face model, a target human face model, and a source virtual character face model;
accessing one or more feature curves marked on the source human face model and the source virtual character face model;
deforming the source virtual character face model based on the source human face model and the target human face model to generate a target virtual character face model, wherein deforming the source virtual character face model comprises:
generating a set of virtual triangles based on vertices of a feature curve of the one or more feature curves;
obtaining a loss function comprising one or more terms, wherein the one or more terms comprise a term defined based on the set of virtual triangles, and wherein the term comprises differences between deformation gradients of the set of virtual triangles and an identity deformation gradient; and
minimizing the loss function to preserve one or more features of the source virtual character face model on the target virtual character face model, wherein the one or more features comprise a local curve feature defined by the feature curve of the one or more feature curves; and
rendering the target virtual character face model.
2 . The method of claim 1 , wherein the local curve feature is one or more of a mouth curve feature or an eye curve feature.
3 . The method of claim 1 , wherein the one or more features comprise a closeness feature defined by a first feature curve and a second feature curve of the one or more feature curves, and the one or more terms comprise a term defined based on a mapping distance between the first feature curve and the second feature curve.
4 . The method of claim 3 , wherein the closeness feature is associated with a mouth or an eye of the virtual character.
5 . The method of claim 1 , wherein the loss function further comprises a term defined based on symmetric pairs of vertices in the source virtual character face model.
6 . The method of claim 1 , wherein the one or more feature curves are defined based on landmark points on the source human face model.
7 . A system comprising:
a non-transitory computer-readable medium; and
a processor communicatively coupled to the non-transitory computer-readable medium, the processor configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to:
access a source human face model, a target human face model, and a source virtual character face model;
access one or more feature curves marked on the source human face model and the source virtual character face model;
deform the source virtual character face model based on the source human face model and the target human face model to generate a target virtual character face model, wherein deforming the source virtual character face model comprises:
generating a set of virtual triangles based on vertices of a feature curve of the one or more feature curves;
obtaining a loss function comprising one or more terms, wherein the one or more terms comprise a term defined based on the set of virtual triangles, and wherein the term comprises differences between deformation gradients of the set of virtual triangles and an identity deformation gradient; and
minimizing the loss function to preserve one or more features of the source virtual character face model on the target virtual character face model, wherein the one or more features comprise a local curve feature defined by the feature curve of the one or more feature curves; and
render the target virtual character face model.
8 . The system of claim 7 , wherein the local curve feature is one or more of a mouth curve feature or an eye curve feature.
9 . The system of claim 7 , wherein the one or more features comprise a closeness feature defined by a first feature curve and a second feature curve of the one or more feature curves, and the one or more terms comprise a term defined based on a mapping distance between the first feature curve and the second feature curve.
10 . The system of claim 9 , wherein the closeness feature is associated with a mouth or an eye of a virtual character.
11 . The system of claim 7 , wherein the loss function further comprises a term defined based on symmetric pairs of vertices in the source virtual character face model.
12 . A non-transitory computer-readable medium comprising processor-executable instructions configured to cause one or more processors to:
access a source human face model, a target human face model, and a source virtual character face model;
access one or more feature curves marked on the source human face model and the source virtual character face model;
deform the source virtual character face model based on the source human face model and the target human face model to generate a target virtual character face model, wherein deforming the source virtual character face model comprises:
generating a set of virtual triangles based on vertices of a feature curve of the one or more feature curves;
obtaining a loss function comprising one or more terms, wherein the one or more terms comprise a term defined based on the set of virtual triangles, and wherein the term comprises differences between deformation gradients of the set of virtual triangles and an identity deformation gradient; and
minimizing the loss function to preserve one or more features of the source virtual character face model on the target virtual character face model, wherein the one or more features comprise a local curve feature defined by the feature curve of the one or more feature curves; and
render the target virtual character face model.
13 . The non-transitory computer-readable medium of claim 12 , wherein the local curve feature is one or more of a mouth curve feature or an eye curve feature.
14 . The non-transitory computer-readable medium of claim 12 , wherein the one or more features comprise a closeness feature defined by a first feature curve and a second feature curve of the one or more feature curves, and the one or more terms comprise a term defined based on a mapping distance between the first feature curve and the second feature curve.