3D object model reconstruction from 2D images

1. A computer-implemented method of reconstructing a three-dimensional model of an object in a three-dimensional scene from a two-dimensional image comprising an image of the object, the method comprising:

determining a size of the object in an image plane of a camera used to capture the image;

converting the size of the object in the image plane into a pixel size based on a camera sampling density of the camera used to capture the image;

determining, from the two-dimensional image, a depth of a key point of the object in the image based on the size of the pixel size of the object using similar triangles;

determining, using the depth of the key point, a three-dimensional position of the key point in the three-dimensional scene;

generating, using a neural network, a three-dimensional representation of the object, the three-dimensional representation comprising a plurality of mesh nodes defined in a coordinate system relative to the key point; and

positioning the three-dimensional representation of the object in the three-dimensional scene based at least in part on the three-dimensional position of the key point,

wherein positioning the three-dimensional representation of the object comprises applying a position dependent rotation to the three-dimensional representation of the object.

2. The method of claim 1 , wherein determining the three-dimensional position of the key point in the three-dimensional scene comprises:

determining, using the same or another neural network, an intersection of a ray connecting a camera that captured the two-dimensional image to the key point of the object with an image plane of the two-dimensional image; and

determining the three-dimensional position of the key point of the object in the coordinate system of the three-dimensional scene based on a world depth and the intersection.

3. The method of claim 1 , wherein determining the three-dimensional position of the key point in the three-dimensional scene is further based on a focal length of a camera that captured the two-dimensional image.

4. The method of claim 1 , wherein the position dependent rotation is a rotation by an angle dependent on a ratio of a two components of the three-dimensional position of the key point, wherein one of the two components is a depth component.

5. The method of claim 1 , further comprising:

projecting the positioned three-dimensional representation into a two-dimensional image plane to generate a projected image of the object;

comparing locations of points in the projected image to the locations of corresponding points in the two-dimensional image; and

updating parameters of the neural network based on the comparing.

6. The method of claim 5 , wherein projecting the positioned three-dimensional representation into a two-dimensional image plane is based at least in part on absolute depth of the key point and depths of the mesh nodes defined by the three-dimensional representation.

7. The method of claim 1 , further comprising training a model using an absolute depth that is a function of the pixel size, the absolute depth being used to determine the depth of the key point.

8. The method of claim 1 , further comprising:

projecting the three-dimensional representation of the object into the image plane to generate a re-projected image;

comparing the re-projected image with the image captured by the camera; and

determining one or more updates to the neural network based on the comparing of the re-projected image with the image captured by the camera.

9. The method of claim 8 , further comprising:

computing a loss based on a position of a particular point of the three-dimensional representation of the object in the re-projected image with a corresponding point in the image captured by the camera.

10. A system comprising:

one or more processors configured to perform operations comprising:

determining a size of an object in an image plane of a camera used to capture an image depicting the object;

converting the size of the object in the image plane into a pixel size based on a camera sampling density of the camera used to capture the image;

determining, from the image, a depth of a key point of the object in the image based on the size of the pixel size of the object using similar triangles;

determining, using the depth of the key point, a three-dimensional position of the key point;

generating, using a neural network, a three-dimensional representation of the object, the three-dimensional representation comprising a plurality of mesh nodes defined in a coordinate system relative to the key point; and

positioning the three-dimensional representation of the object based at least in part on the three-dimensional position of the key point,

wherein positioning the three-dimensional representation of the object comprises applying a position dependent rotation to the three-dimensional representation of the object.

11. The system of claim 10 , wherein determining the three-dimensional position of the key point comprises:

determining, using the same or another neural network, an intersection of a ray connecting a camera that captured the two-dimensional image to the key point of the object with an image plane of the two-dimensional image; and

determining the three-dimensional position of the key point of the object based on a world depth and the intersection.

12. The system of claim 10 , wherein determining the three-dimensional position of the key point is further based on a focal length of a camera that captured the two-dimensional image.

13. The system of claim 10 , wherein the position dependent rotation is a rotation by an angle dependent on a ratio of a two components of the three-dimensional position of the key point, wherein one of the two components is a depth component.

14. The system of claim 10 , the operations further comprising:

projecting the positioned three-dimensional representation into a two-dimensional image plane to generate a projected image of the object;

comparing locations of points in the projected image to the locations of corresponding points in the two-dimensional image; and

updating parameters of the neural network based on the comparing.

15. The system of claim 14 , wherein projecting the positioned three-dimensional representation into a two-dimensional image plane is based at least in part on absolute depth of the key point and depths of the mesh nodes defined by the three-dimensional representation.

16. The system of claim 10 , the operations further comprising training a model using an absolute depth that is a function of the pixel size, the absolute depth being used to determine the depth of the key point.

17. The system of claim 10 , the operations further comprising:

projecting the three-dimensional representation of the object into the image plane to generate a re-projected image;

comparing the re-projected image with the image captured by the camera; and

determining one or more updates to the neural network based on the comparing of the re-projected image with the image captured by the camera.

18. The system of claim 17 , the operations further comprising:

computing a loss based on a position of a particular point of the three-dimensional representation of the object in the re-projected image with a corresponding point in the image captured by the camera.

19. A non-transitory computer readable medium comprising computer readable instructions that, when executed by one or more processors, configure the one or more processors to perform operations comprising:

one or more processors configured to perform operations comprising:

determining a size of an object in an image plane of a camera used to capture an image depicting the object;

converting the size of the object in the image plane into a pixel size based on a camera sampling density of the camera used to capture the image;

determining, from the image, a depth of a key point of the object in the image based on the size of the pixel size of the object using similar triangles;

determining, using the depth of the key point, a three-dimensional position of the key point;

generating, using a neural network, a three-dimensional representation of the object, the three-dimensional representation comprising a plurality of mesh nodes defined in a coordinate system relative to the key point; and

positioning the three-dimensional representation of the object based at least in part on the three-dimensional position of the key point,

wherein positioning the three-dimensional representation of the object comprises applying a position dependent rotation to the three-dimensional representation of the object.

20. The non-transitory computer readable medium of claim 19 , wherein determining the three-dimensional position of the key point comprises:

determining, using the same or another neural network, an intersection of a ray connecting a camera that captured the two-dimensional image to the key point of the object with an image plane of the two-dimensional image; and

determining the three-dimensional position of the key point of the object based on a world depth and the intersection.

21. The non-transitory computer readable medium of claim 19 , wherein determining the three-dimensional position of the key point is further based on a focal length of a camera that captured the two-dimensional image.