Techniques for generating dubbed media content items
In various embodiments, a dubbing application performs three-dimensional (3D) tracking of (1) the face of an actor within video frames of a first media content item to generate 3D geometry representing the face of the actor, and (2) the face of a dubber within video frames of a second media content item to generate 3D geometry representing the face of the dubber. The dubbing application also tracks the texture and lighting of the face of the actor in the first media content item. The dubbing application aligns the 3D geometry of the face of the dubber with the 3D geometry of the face of the actor. Then, the dubbing application performs neural rendering to generate dubbed video frames using a trained machine learning model, the aligned 3D geometry of the dubber, the texture and lighting of the face of the actor, and the video frames of the first media content.
1 . A computer-implemented method for generating dubbed media content items, the method comprising:
generating first three-dimensional (3D) geometry, a texture map, and a lighting map based on a face of an actor included in a first video frame of a first media content item;
generating second 3D geometry based on a face of a dubber included in a second video frame of a second media content item;
performing one or more operations that align the second 3D geometry with the first 3D geometry to generate an aligned second 3D geometry;
performing one or more operations to convert the lighting map to a neural lighting; and
performing one or more operations via one or more trained machine learning models to render a third video frame based on the aligned second 3D geometry, the texture map, the neural lighting, and the first video frame of the first media content item.
2 . The computer-implemented method of claim 1 , wherein generating the first 3D geometry comprises:
detecting a plurality of landmarks on the face of the actor included in the first video frame;
performing one or more operations to fit an intermediate 3D geometry to the plurality of landmarks; and
performing one or more optimization operations to update the intermediate 3D geometry based on the first video frame and one or more loss functions.
3 . The computer-implemented method of claim 2 , wherein the one or more loss functions penalize a difference between a mapping of the intermediate 3D geometry to a canonical space and one or more other mappings of one or more other 3D geometries associated with the face of the actor to the canonical space.
4 . The computer-implemented method of claim 2 , wherein the one or more loss functions penalize one or more differences between one or more landmarks on lips of the actor included in the first video frame and one or more corresponding landmarks on lips associated with the intermediate 3D geometry.
5 . The computer-implemented method of claim 2 , wherein the one or more loss functions penalize one or more differences between one or more landmarks on teeth of the actor included in the first video frame and one or more corresponding landmarks on teeth associated with the intermediate 3D geometry.
6 . The computer-implemented method of claim 2 , wherein the one or more loss functions penalize a difference between a degree to which a mouth associated with the intermediate 3D geometry is closed and a degree to which a detected mouth of the face of the actor included in the first video frame is closed.
7 . The computer-implemented method of claim 2 , wherein the one or more loss functions penalize a difference between the plurality of landmarks on the face of the actor included in the first video frame and a plurality of corresponding landmarks associated with the intermediate 3D geometry.
8 . The computer-implemented method of claim 1 , wherein the texture map and the lighting map are generated based on a loss function that penalizes a difference between the first video frame and a fourth video frame that has been rendered using the texture map and the lighting map.
9 . The computer-implemented method of claim 1 , wherein performing the one or more operations that align the second 3D geometry with the first 3D geometry comprises:
performing one or more operations to align a nose position and a mouth position associated with the second 3D geometry with a nose position and a mouth position associated with the first 3D geometry;
performing one or more operations to equalize a scale of one or more expressions associated with the second 3D geometry with a scale of one or more expressions associated with the first 3D geometry; and
performing one or more optimization operations to determine the one or more expressions associated with the second 3D geometry when combining a bottom portion of the second 3D geometry with a top portion of the first 3D geometry.
10 . The computer-implemented method of claim 1 , wherein performing the one or more operations to render the third video frame comprises:
performing one or more operations to convert the texture map to a neural texture;
and
processing, using a first trained machine learning model, the aligned second 3D geometry, the first video frame, a mask indicating one or more regions of the first video frame to be inpainted, and a combination of the neural texture and the neural lighting to generate the third video frame.
11 . One or more non-transitory computer-readable media storing instructions that, when executed by at least one processor, cause the at least one processor to perform steps comprising:
generating first three-dimensional (3D) geometry, a texture map, and a lighting map based on a face of an actor included in a first video frame of a first media content item;
generating second 3D geometry based on a face of a dubber included in a second video frame of a second media content item;
performing one or more operations that align the second 3D geometry with the first 3D geometry to generate an aligned second 3D geometry;
performing one or more operations to convert the lighting map to a neural lighting; and
performing one or more operations via one or more trained machine learning models to render a third video frame based on the aligned second 3D geometry, the texture map, the neural lighting, and the first video frame of the first media content item.
12 . The one or more non-transitory computer-readable media of claim 11 , wherein generating the first 3D geometry comprises:
detecting a plurality of landmarks on the face of the actor included in the first video frame;
performing one or more operations to fit an intermediate 3D geometry to the plurality of landmarks; and
performing one or more optimization operations to update the intermediate 3D geometry based on the first video frame and one or more loss functions.
13 . The one or more non-transitory computer-readable media of claim 12 , wherein the one or more loss functions penalize a difference between a mapping of the intermediate 3D geometry to a canonical space and one or more other mappings of one or more other 3D geometries associated with the face of the actor to the canonical space.
14 . The one or more non-transitory computer-readable media of claim 12 , wherein the one or more loss functions penalize one or more differences between one or more landmarks on at least one of lips or teeth of the actor included in the first video frame and one or more corresponding landmarks on at least one of lips or teeth associated with the intermediate 3D geometry.
15 . The one or more non-transitory computer-readable media of claim 11 , wherein performing the one or more operations that align the second 3D geometry with the first 3D geometry comprises:
performing one or more operations to align a nose position and a mouth position associated with the second 3D geometry with a nose position and a mouth position associated with the first 3D geometry;
performing one or more operations to equalize a scale of one or more expressions associated with the second 3D geometry with a scale of one or more expressions associated with the first 3D geometry; and
performing one or more optimization operations to determine the one or more expressions associated with the second 3D geometry when combining a bottom portion of the second 3D geometry with a top portion of the first 3D geometry.
16 . The one or more non-transitory computer-readable media of claim 11 , wherein performing the one or more operations to render the third video frame comprises:
performing one or more operations to convert the texture map to a neural texture;
and
processing, using a first trained machine learning model, the aligned second 3D geometry, the first video frame, a mask indicating one or more regions of the first video frame to be inpainted, and a combination of the neural texture and the neural lighting to generate the third video frame.
17 . The one or more non-transitory computer-readable media of claim 16 , wherein performing the one or more operations to render the third video frame further comprises performing one or more operations to crop and center the face of the actor in the first video frame based on the first 3D geometry.
18 . The one or more non-transitory computer-readable media of claim 16 , wherein the mask is applied to one or more feature spaces of the first trained machine learning model.
19 . The one or more non-transitory computer-readable media of claim 16 , wherein the first trained machine learning model comprises an encoder network and a decoder network.
20 . A system, comprising:
a memory storing instructions; and
a processor that is coupled to the memory and, when executing the instructions, is configured to perform the steps of:
generating first three-dimensional (3D) geometry, a texture map, and a lighting map based on a face of an actor included in a first video frame of a first media content item,
generating second 3D geometry based on a face of a dubber included in a second video frame of a second media content item,
performing one or more operations that align the second 3D geometry with the first 3D geometry to generate an aligned second 3D geometry,
performing one or more operations to convert the lighting map to a neural lighting, and
performing one or more operations via one or more trained machine learning models to render a third video frame based on the aligned second 3D geometry, the texture map, the neural lighting, and the first video frame of the first media content item.