Deformable neural radiance field for editing facial pose and facial expression in neural 3D scenes
A scene modeling system receives a video including a plurality of frames corresponding to views of an object and a request to display an editable three-dimensional (3D) scene that corresponds to a particular frame of the plurality of frames. The scene modeling system applies a scene representation model to the particular frame, and includes a deformation model configured to generate, for each pixel of the particular frame based on a pose and an expression of the object, a deformation point using a 3D morphable model (3DMM) guided deformation field. The scene representation model includes a color model configured to determine, for the deformation point, color and volume density values. The scene modeling system receives a modification to one or more of the pose or the expression of the object including a modification to a location of the deformation point and renders an updated video based on the received modification.
1. A method, comprising:
receiving, at a scene modeling system, a video including a plurality of frames corresponding to a plurality of views of an object and a request to display an editable three-dimensional (3D) scene that includes the object and that corresponds to a particular frame of the plurality of frames;
generating, by the scene modeling system, the editable 3D scene by applying a scene representation model to the particular frame, wherein the scene representation model comprises:
a deformation model configured to generate a 3D morphable model (“3DMM”) guided deformation field based on a 3DMM deformation field and a residual predicted by the deformation model, and further generate, for each pixel of the particular frame and based on a pose and an expression of the object, a deformation point using the 3DMM guided deformation field, and
a color model configured to determine, for the deformation point and using a volume rendering process, a color value and a volume density value; and
providing, by the scene modeling system, the editable 3D scene to a computing device executing a scene modeling application configured to generate a modified video using the editable 3D scene, wherein the scene modeling application generates the modified video by:
receiving a modification to one or more of the pose or the expression of the object including at least a modification to a location of the deformation point,
receiving a modification to a view of the editable 3D scene, wherein the modification to the view comprises one or more of a change to a camera position or to a camera orientation within the editable 3D scene,
rendering an updated editable 3D scene based on the received modification to the one or more of the pose or the expression of the object and by changing the view of the editable 3D scene to the modified view, and
generating the modified video including an updated frame to replace the particular frame, the updated frame generated based on the updated editable 3D scene.
2. The method of claim 1 , wherein the edit to the object comprises a change to one or more of the pose or the expression.
3. The method of claim 1 , wherein the pose of the object represents an orientation of the object with respect to a default pose and wherein the edit to the pose of the object corresponds a change in the orientation of the object.
4. The method of claim 1 , wherein the expression of the object corresponds to a first semantic category and wherein the edit to the expression of the object comprises a selection of a second semantic category.
5. The method of claim 1 , wherein the pose and the expression are extracted from the particular frame using a detailed expression capture and animation (“DECA”) method.
6. The method of claim 1 , wherein deforming the point using the 3DMM guided deformation field comprises transforming the point to a canonical space, wherein the color model is applied to the transformed point.
7. The method of claim 1 , wherein each of the deformation model and the color model comprise a multilayer perceptron model, wherein the deformation model is configured to generate the 3DMM guided deformation field, residual, and deformation points in a first stage, and the color model is configured to determine, in a second stage, a color value and volume density value based on the deformation point generated in the first stage, and further based on the pose and expression of the object.
8. A system, comprising:
a memory component; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
generating, for a particular frame of a video, an editable 3D scene that includes an object that is depicted in the particular frame of the video by applying a scene representation model to the particular frame, wherein the video includes a plurality of frames corresponding to a plurality of views of an object including the particular frame, wherein the scene representation model comprises:
a deformation model configured to generate a 3D morphable model (“3DMM”) guided deformation field based on a 3DMM deformation field, and a residual predicted by the deformation model, and further generate, for each pixel of the particular frame and based on a pose and an expression of the object, a deformation point using the 3DMM guided deformation field, and
a color model configured to determine, for the deformation point and using a volume rendering process, a color value and a volume density value; and
providing the editable 3D scene to a computing device executing a scene modeling application configured to generate a modified video using the editable 3D scene, wherein the scene modeling application generates the modified video by:
receiving a modification to one or more of the pose or the expression of the object including at least a modification to a location of the deformation point,
receiving a modification to a view of the editable 3D scene, wherein the modification to the view comprises one or more of a change to a camera position or to a camera orientation within the editable 3D scene,
rendering an updated editable 3D scene based on the received modification to the one or more of the pose or the expression of the object and by changing the view of the editable 3D scene to the modified view, and
generating a modified video including an updated frame to replace the particular frame, the updated frame generated based on the updated editable 3D scene.
9. The system of claim 8 , wherein the edit to the object comprises a change to one or more of the pose or the expression.
10. The system of claim 8 , wherein the pose of the object represents an orientation of the object with respect to a default pose and wherein the edit to the pose of the object corresponds a change in the orientation of the object.
11. The system of claim 8 , wherein the expression of the object corresponds to a first semantic category and wherein the edit to the expression of the object comprises a selection of a second semantic category.
12. The system of claim 8 , wherein deforming the point using the 3DMM guided deformation field comprises transforming the point to a canonical space, wherein the color model is applied to the transformed point.
13. The system of claim 8 , wherein each of the deformation model and the color model comprise a multilayer perceptron model, wherein the deformation model is configured to generate the 3DMM guided deformation field, residual, and deformation points in a first stage, and the color model is configured to determine, in a second stage, a color value and volume density value based on the deformation point generated in the first stage, and further based on the pose and expression of the object.
14. A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
generating, for a particular frame of a video, an editable 3D scene that includes an object that is depicted in the particular frame of the video by applying a scene representation model to the particular frame, wherein the video includes a plurality of frames corresponding to a plurality of views of an object including the particular frame, wherein the scene representation model comprises:
a deformation model configured to generate a 3D morphable model (“3DMM”) guided deformation field based on a 3DMM deformation field, and a residual predicted by the deformation model, and further generate, for each pixel of the particular frame and based on a pose and an expression of the object, a deformation point using the 3DMM guided deformation field, and
a color model configured to determine, for the deformation point and using a volume rendering process, a color value and a volume density value;
receiving a modification to one or more of the pose or the expression of the object including at least a change in location of the deformation point;
receiving a modification to a view of the editable 3D scene, wherein the modification to the view comprises one or more of a change to a camera position or to a camera orientation within the editable 3D scene;
rendering an updated editable 3D scene based on the received modification to the one or more of the pose or the expression of the object and by changing the view of the editable 3D scene to the modified view; and
generating a modified video including an updated frame to replace the particular frame, the updated frame generated based on the updated editable 3D scene.
15. The non-transitory computer-readable medium of claim 14 , wherein the edit to the object comprises a change to one or more of the pose or the expression.
16. The non-transitory computer-readable medium of claim 14 , wherein deforming the point using the 3DMM guided deformation field comprises transforming the point to a canonical space, wherein the color model is applied to the transformed point.
17. The non-transitory computer-readable medium of claim 14 , wherein each of the deformation model and the color model comprise a multilayer perceptron model, wherein the deformation model is configured to generate the 3DMM guided deformation field, residual, and deformation points in a first stage, and the color model is configured to determine, in a second stage, a color value and volume density value based on the deformation point generated in the first stage, and further based on the pose and expression of the object.