Learning articulated shape reconstruction from imagery
A computing system and method can be used to render a 3D shape from one or more images. In particular, the present disclosure provides a general pipeline for learning articulated shape reconstruction from images (LASR). The pipeline can reconstruct rigid or nonrigid 3D shapes. In particular, the pipeline can automatically decompose non-rigidly deforming shapes into rigid motions near rigid-bones. This pipeline incorporates an analysis-by-synthesis strategy and forward-renders silhouette, optical flow, and color images which can be compared against the video observations to adjust the internal parameters of the model. By inverting a rendering pipeline and incorporating optical flow, the pipeline can recover a mesh of a 3D model from the one or more images input by a user.
1 . A computer-implemented method for determining 3D object shape from imagery, the method comprising:
obtaining, by a computing system comprising one or more computing devices, an input image that depicts an object and a current mesh model of the object, wherein the current mesh model of the object comprises a plurality of vertices, a plurality of joints, and a plurality of blend skinning weights for the plurality of joints relative to the plurality of vertices;
processing, by the computing system, the input image with a camera model to obtain camera parameters, articulation parameters, and object deformation data for the input image, wherein the camera parameters describe a camera pose for the input image, wherein the articulation parameters are based on the plurality of vertices, the plurality of joints, and the plurality of blend skinning weights, wherein the object deformation data describes one or more deformations of the current mesh model relative to a shape of the object shown in the input image, and wherein the processing comprises linear blend skinning based on the articulation parameters;
differentiably rendering, by the computing system, a rendered image of the object based on the camera parameters, the articulation parameters, the object deformation data, and the current mesh model;
evaluating, by the computing system, a loss function that compares one or more characteristics of the input image of the object with one or more characteristics of the rendered image of the object; and
modifying, by the computing system, one or more values of one or both of the camera model and the current mesh model based on a gradient of the loss function.
2 . The computer-implemented method of claim 1 , wherein evaluating the loss function comprises:
determining a first flow for the input image;
determining a second flow for the rendered image; and
evaluating the loss function based at least in part on a comparison of the first flow and the second flow.
3 . The computer-implemented method of claim 1 , wherein evaluating the loss function comprises:
determining a first silhouette for the input image;
determining a second silhouette for the rendered image; and
evaluating the loss function based at least in part on a comparison of the first silhouette and the second silhouette.
4 . The computer-implemented method of claim 1 , wherein evaluating the loss function comprises evaluating the loss function based at least in part on a comparison of first texture data associated with the input image and second texture data associated with the rendered image.
5 . The computer-implemented method of claim 1 , wherein the mesh model comprises a triangular mesh model.
6 . The computer-implemented method of claim 1 , wherein the camera parameters describes an object-to-camera transformation for the input image.
7 . The computer-implemented method of claim 1 , wherein the camera model comprises a convolutional neural network.
8 . The computer-implemented method of claim 1 , wherein the camera parameters further describe a focal length for the input image.
9 . The computer-implemented method of claim 1 , wherein the mesh model is initialized to a subdivided icosahedron projected to a sphere.
10 . The computer-implemented method of claim 1 , wherein one or both of the camera model and the current mesh model comprise both the camera model and the current mesh model.
11 . The computer-implemented method of claim 1 , wherein the plurality of joints and the plurality of blend skinning weights are learnable.
12 . The computer-implemented method of claim 1 , wherein differentiably rendering the rendered image of the object based on the camera parameters, the object deformation data, and the current mesh model comprises rendering the current mesh model deformed according to the object deformation data and from the camera pose according to the camera parameters.
13 . The computer-implemented method of claim 1 , further comprising performing the method of claim 1 on multiple images that depict multiple objects to build a library of shape models from the multiple images.
14 . The computer-implemented method of claim 1 , wherein obtaining the input image that depicts the object comprises manually choosing a canonical image frame from a video.
15 . The computer-implemented method of claim 1 , wherein obtaining the input image that depicts the object comprises:
selecting a number of candidate frames;
evaluating a loss for each of the candidate frames; and
selecting the candidate frame with a lowest final loss as a canonical frame.
16 . A computer system, comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store:
a machine-learned mesh model for an object, wherein the machine-learned mesh model for the object comprises a plurality of vertices, a plurality of joints, and a plurality of blend skinning weights for the plurality of joints relative to the plurality of vertices, wherein the machine-learned mesh model for the object has been learned jointly with a machine-learned camera model by minimizing a loss function that evaluates a difference between one or more input images of the object and one or more rendered images of the object, the one or more rendered images comprising images rendered based on the machine-learned mesh model, camera parameters, and linear blend skimming based on articulation parameters, wherein the articulation parameters and the camera parameters are generated by the machine-learned camera model from the one or more input images, and wherein the articulation parameters are based on the plurality of vertices, the plurality of joints, and the plurality of blend skinning weights; and
instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations, comprising:
receiving an additional set of camera parameters; and
rendering an additional rendered image of the object based on the machine-learned mesh model and the additional set of camera parameters.
17 . The computing system of claim 16 , wherein the loss function compares a first flow for each input image with a second flow for each rendered image.
18 . The computing system of claim 16 , wherein the loss function compares a first silhouette for each input image with a second silhouette for each rendered image.
19 . The computing system of claim 16 , wherein the loss function compares a first texture for each input image with a second texture for each rendered image.
20 . One or more non-transitory computer-readable media that collectively store instructions that, when executed by a computing system comprising one or more computing devices, cause the computing to perform operations, the operations comprising:
obtaining, by the computing system, an input image that depicts an object and a current mesh model of the object, wherein the current mesh model comprises a plurality of vertices, a plurality of joints, and a plurality of blend skinning weights for the plurality of joints relative to the plurality of vertices;
processing, by the computing system, the input image with a camera model to obtain camera parameters, articulation parameters, and object deformation data for the input image, wherein the camera parameters describe a camera pose for the input image, wherein the articulation parameters are based on the plurality of vertices, the plurality of joints, and the plurality of blend skinning weights, wherein the object deformation data describes one or more deformations of the current mesh model relative to a shape of the object shown in the image, and wherein the processing comprises linear blend skinning based on the articulation parameters;
differentiably rendering, by the computing system, a rendered image of the object based on the camera parameters, the articulation parameters, the object deformation data, and the current mesh model;
evaluating, by the computing system, a loss function that compares one or more characteristics of the input image of the object with one or more characteristics of the rendered image of the object; and
modifying, by the computing system, one or more values of one or both of the camera model and the current mesh model based on a gradient of the loss function.