Generating three-dimensional object models from two-dimensional images
This specification discloses methods and systems for generating three-dimensional models of deformable objects from two-dimensional images. According to one aspect of this disclosure, there is described a computer implemented method for generating a three dimensional model of deformable object from a two-dimensional image. The method comprises: receiving, as input to an embedding neural network, the two-dimensional image, wherein the two dimensional image comprises an image of an object; generating, using the embedding neural network, an embedded representation of a two-dimensional image; inputting the embedded representation into a learned decoder model; and generating, using the learned decoder model, parameters of the three dimensional model of the object from the embedded representation.
1. A computer implemented method, comprising:
receiving, as input to an embedding neural network that comprises one or more encoder parameters, a two-dimensional training image, wherein the two-dimensional training image comprises an object and is associated with a predetermined three-dimensional model that comprises vertices;
generating, using the embedding neural network, a first embedded representation of the two-dimensional training image;
inputting the first embedded representation into a decoder model, the decoder model comprising a regression model that comprises one or more decoder parameters;
generating, using the decoder model, a generated three-dimensional model of the object based on the first embedded representation, the generated three-dimensional model comprising vertices;
comparing, using a loss function, the generated three-dimensional model of the object to the predetermined three-dimensional model associated with the two-dimensional training image;
updating the one or more encoder parameters and the one or more decoder parameters of the decoder model in dependence on the comparing: receiving, as input to the embedding neural network, a two-dimensional object image, wherein the two-dimensional object image comprises an image of the object;
generating, using the embedding neural network, a second embedded representation of a two-dimensional object image;
inputting the second embedded representation of the two-dimensional object image into the decoder model that comprises the regression model; and
generating, using the decoder model, a resulting three-dimensional model of the object from the second embedded representation, the resulting three-dimensional model comprising a plurality of nodes in a mesh;
wherein the loss function comprises a vertex term and an edge term, the vertex term comprising first differences between vertices of the generated three-dimensional model and vertices of the predetermined three-dimensional model, the edge term comprising second differences between edge lengths of the generated three-dimensional model and edge lengths of the predetermined three-dimensional model.
2. The method of claim 1 , wherein the generating the generated three-dimensional model of the object, the comparing the generated three-dimensional model of the object to the predetermined three-dimensional model, and the updating the one or more encoder parameters and the one or more decoder parameters of the decoder model in dependence on the comparing are repeated until a threshold condition is satisfied.
3. The method of claim 1 , wherein the vertices of the predetermined three-dimensional model of the object associated with the two-dimensional training image are derived by fitting a three-dimensional object model to the two-dimensional training image.
4. The method of claim 1 , wherein the decoder model comprises a decoder neural network comprising one or more two-dimensional convolutional layers, and wherein the generated three-dimensional model corresponds to a hand, the generated three-dimensional model comprises three-dimensional co-ordinates of the plurality of nodes of the mesh.
5. The method of claim 4 , wherein the decoder neural network comprises one or more upscaling layers.
6. The method of claim 4 , wherein initial layers of the decoder neural network comprise a fully connected layer and a reshaping layer.
7. The method of claim 1 , wherein the decoder model comprises a linear layer and wherein the generated three-dimensional model corresponds to a hand, the generated three-dimensional model comprises three-dimensional co-ordinates of the plurality of nodes of the mesh.
8. The method of claim 1 , further comprising:
generating a training dataset for training the decoder model to generate a generated three-dimensional model of a deformable object from the two-dimensional object image, the generated three-dimensional model comprising generated parameters that comprise a plurality of nodes in a mesh; the generating the training dataset comprising:
extracting a plurality of two-dimensional training images from one or more sections of video data, each two-dimensional training image of the plurality of two-dimensional training images comprising one or more deformable objects;
for the each two-dimensional training image in the plurality of two-dimensional training images:
identifying, using a keypoint identification model, a plurality of object keypoints in the each two-dimensional training image;
determining parameters of a three-dimensional parametric model, the parameters corresponding to the deformable object in the each two-dimensional training image, wherein determining the parameters comprises fitting the three-dimensional parametric model of the object to the each two-dimensional training image using the identified object keypoints and corresponding object keypoints in the three-dimensional parametric model; and
storing, in a computer memory, the each two-dimensional training image and the corresponding parameters of the three-dimensional parametric model.
9. The method of claim 8 , wherein the one or more sections of video data comprises one or more sections of video of use of sign language.
10. The method of claim 8 , wherein fitting the three-dimensional parametric model of the object to the each two-dimensional image comprises applying an iterative optimization procedure to an objective function,
wherein the objective function comprises a reprojection term comparing locations of object keypoints identified in the each two-dimensional training image with locations of corresponding object keypoints in a two-dimensional projection of the three-dimensional parametric model.
11. The method of claim 10 , wherein the reprojection term comprises a mask that applies a weighting to the corresponding object keypoints in the two-dimensional projection.
12. The method of claim 10 , wherein the objective function further comprises a length term comparing distances between object keypoints identified in the each two-dimensional training image and distances between corresponding object keypoints in a two-dimensional projection of the three-dimensional parametric model.
13. The method of claim 10 , wherein the objective function further comprises a regularization term penalizing deviations from a predefined mean object pose.
14. The method of claim 8 , wherein the three-dimensional parametric model of the object is a mesh model parametrized by parameters comprising one or more shape parameters and/or one or more pose parameters.
15. The method of claim 13 , wherein the parameters of the three-dimensional parametric model of the object further comprise one or more camera parameters.
16. The method of claim 14 , wherein the corresponding object keypoints in the three-dimensional parametric model are identified from the mesh model using a sparse regression model.
17. A computing apparatus, comprising:
a processor;
a memory, the memory comprising computer readable instructions that, when executed by the processor, cause the computing apparatus to perform operations comprising:
receiving, as input to an embedding neural network that comprises one or more encoder parameters, a two-dimensional training image, wherein the two-dimensional training image comprises an object and is associated with a predetermined three-dimensional model that comprises vertices;
generating, using the embedding neural network, a first embedded representation of the two-dimensional training image;
inputting the first embedded representation into a decoder model, the decoder model comprising a regression model that comprises one or more decoder parameters;
generating, using the decoder model, a generated three-dimensional model of the object based on the first embedded representation, the generated three-dimensional model comprising vertices;
comparing, using a loss function, the generated three-dimensional model of the object to the predetermined three-dimensional model associated with the two-dimensional training image;
updating the one or more encoder parameters and the one or more decoder parameters of the decoder model in dependence on the comparing;
receiving, as input to the embedding neural network, a two-dimensional object image, wherein the two-dimensional object image comprises an image of the object;
generating, using the embedding neural network, a second embedded representation of a two-dimensional object image;
inputting the second embedded representation of the two-dimensional object image into the decoder model that comprises the regression model; and
generating, using the decoder model, a resulting three-dimensional model of the object from the second embedded representation, the resulting three-dimensional model comprising a plurality of nodes in a mesh;
wherein the loss function comprises a vertex term and an edge term, the vertex term comprising first differences between vertices of the generated three-dimensional model and vertices of the predetermined three-dimensional model, the edge term comprising second differences between edge lengths of the generated three-dimensional model and edge lengths of the predetermined three-dimensional model.
18. The computing apparatus of claim 17 , wherein the generating the generated three-dimensional model of the object, the comparing the generated three-dimensional model of the object to the predetermined three-dimensional model, and the updating the one or more encoder parameters and the one or more decoder parameters of the decoder model in dependence on the comparing are repeated until a threshold condition is satisfied.
19. A non-transitory computer readable storage medium comprising instructions that when executed configure hardware processing circuitry to perform operations comprising:
receiving, as input to an embedding neural network that comprises one or more encoder parameters, a two-dimensional training image, wherein the two-dimensional training image comprises an object and is associated with a predetermined three-dimensional model that comprises vertices;
generating, using the embedding neural network, a first embedded representation of the two-dimensional training image;
inputting the first embedded representation into a decoder model, the decoder model comprising a regression model that comprises one or more decoder parameters;
generating, using the decoder model, a generated three-dimensional model of the object based on the first embedded representation, the generated three-dimensional model comprising vertices;
comparing, using a loss function, the generated three-dimensional model of the object to the predetermined three-dimensional model associated with the two-dimensional training image;
updating the one or more encoder parameters and the one or more decoder parameters of the decoder model in dependence on the comparing;
receiving, as input to the embedding neural network, a two-dimensional object image, wherein the two-dimensional object image comprises an image of the object;
generating, using the embedding neural network, a second embedded representation of a two-dimensional object image;
inputting the second embedded representation of the two-dimensional object image into the decoder model that comprises the regression model; and
generating, using the decoder model, a resulting three-dimensional model of the object from the second embedded representation, the resulting three-dimensional model comprising a plurality of nodes in a mesh;
wherein the loss function comprises a vertex term and an edge term, the vertex term comprising first differences between vertices of the generated three-dimensional model and vertices of the predetermined three-dimensional model, the edge term comprising second differences between edge lengths of the generated three-dimensional model and edge lengths of the predetermined three-dimensional model.