Annotation of 3D models with signs of use visible in 2D images
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for annotation of 3D models with signs of use that are visible in 2D images. In one aspect, methods are performed by data processing apparatus. The methods can include projecting signs of use in a relatively larger field of view image of an instance of an object onto a 3D model of the object based on a pose of the instance in the relatively larger field of view image, and estimating a relative pose of the instance of the object in a relatively smaller field of view image based on matches between the signs of use in the relatively larger field of view image and the same signs of use in the relatively smaller field of view image.
1. A method performed by data processing apparatus, the method comprising
projecting signs of use in a relatively larger field of view image of an instance of an object onto a 3D model of the object based on a pose of the instance in the relatively larger field of view image; and
estimating a relative pose of the instance of the object in a relatively smaller field of view image based on matches between the signs of use in the relatively larger field of view image and the same signs of use in the relatively smaller field of view image.
2. The method of claim 1 , further comprising:
computing hypothetical positions of signs of use in the relatively smaller field of view image using the estimated pose and the projection of signs of use onto the 3Dmodel;
comparing the hypothetical positions with actual positions of the signs of use in the relatively smaller field of view image; and
identifying, based on the comparison, a subset of the signs of use that are improperly projected onto the 3D model.
3. The method of claim 2 , further comprising:
deforming the 3D model using a second subset of the signs of use that are properly projected onto the 3D model.
4. The method of claim 2 , further comprising:
projecting an improperly projected of the signs of use onto the 3D model of the object based on the relative pose of the instance of the object in a relatively smaller field of view image.
5. The method of claim 4 , wherein projecting the improperly projected of the signs of use onto the 3D model comprises:
identifying a region of the relatively smaller field of view image that includes a first of the improperly projected of the signs of use;
matching the first of the improperly projected of the signs of use to a first of the signs of use in the relatively larger field of view image; and
projecting the first of the signs of use in the relatively larger field of view image onto the 3D model of the object.
6. The method of claim 2 , wherein the improperly projected subset of the signs of use are identified based on a positional deviation between the hypothetical positions and the actual positions of the signs of use in the relatively smaller field of view image.
7. The method of claim 2 , wherein the method comprises:
filtering the subset of the signs of use that are improperly projected onto the 3D model from the matches to establish a proper subset of the matches; and
again estimating the relative pose of the instance of the object in the relatively smaller field of view image based on subset of the matches.
8. The method of claim 1 , wherein projecting the signs of use in the relatively larger field of view image onto the 3D model comprises determining a pose of the relatively larger field of view image.
9. The method of claim 1 , further comprising:
determining a dominant color of the instance of the object in the relatively smaller field of view image;
identifying regions in the relatively larger field of view image, in the relatively smaller field of view image, or in both the relatively larger field of view image and the relatively smaller field of view image that deviate from the dominant color; and
matching the identified regions to match the signs of use in the relatively larger field of view image and the signs of use in the relatively smaller field of view image.
10. The method of claim 1 , further comprising:
determining a deviation from ideality in the instance of the object in the relatively smaller field of view image; and
matching the deviation in the relatively smaller field of view image to the relatively larger field of view image to match the signs of use in the relatively larger field of view image and the signs of use in the relatively smaller field of view image.
11. The method of claim 1 , comprising:
estimating relative poses of the instance of the object in a plurality of relatively smaller field of view images; and
computing hypothetical positions of the same signs of use in the relatively smaller field of view images using the estimated poses.
12. A method performed by data processing apparatus, the method comprising annotating a 3D model of an object with signs of use from two or more 2D images of an instance of the object, wherein annotating the 3D model comprises:
receiving the 3D model and the 2D images, wherein a first of the 2D images is a relatively larger field of view image of the instance and a second of the 2D images is a relatively smaller field of view image of the instance;
matching signs of use that are visible in the relatively larger field of view and relatively smaller field of view images;
projecting the signs of use in the relatively larger field of view image onto the 3D model of the object;
estimating a pose of the instance in the relatively smaller field of view image using the projection of signs of use onto the 3D model and the matched signs of use in the relatively larger and relatively smaller field of view images;
computing hypothetical positions of signs of use in the relatively smaller field of view image using the estimated pose and the projection of signs of use onto the 3D model;
comparing the hypothetical positions with actual positions of the signs of use in the relatively smaller field of view image to identify improperly projected of the signs of use; and
eliminating the improperly projected of the signs of use from the projections onto the 3D model of the object.
13. The method of claim 12 , further comprising:
projecting the improperly projected of the signs of use onto the 3D model of the object based on the relative pose of the instance of the object in a relatively smaller field of view image.
14. The method of claim 13 , wherein projecting the improperly projected of the signs of use onto the 3D model comprises:
identifying a region of the relatively smaller field of view image that includes a first of the improperly projected of the signs of use;
matching the first of the improperly projected of the signs of use to a first of the signs of use in the relatively larger field of view image; and
projecting the first of the signs of use in the relatively larger field of view image onto the 3D model of the object.
15. The method of claim 12 , wherein the improperly projected subset of the signs of use are identified based on a positional deviation between the hypothetical positions and the actual positions of the signs of use in the relatively smaller field of view image.
16. The method of claim 12 , wherein matching the signs of use comprises:
determining the dominant color of the instance of the object in the relatively smaller field of view image;
identifying regions in the relatively larger field of view image, in the relatively smaller field of view image, or in both the relatively larger field of view image and the relatively smaller field of view image that deviate from the dominant color.
17. The method of claim 12 , wherein matching the signs of use comprises:
determining a deviation from ideality in the instance of the object in the relatively smaller field of view image; and
matching the deviation in the relatively smaller field of view image to the relatively larger field of view image.
18. The method of claim 12 , further comprising:
deforming the 3D model using properly projected of the signs of use from the projections onto the 3D model of the object.
19. At least one computer-readable storage medium encoded with executable instructions that, when executed by at least one processor, cause the at least one processor to perform operations for annotating a 3D model of an object with signs of use from two or more 2D images of an instance of the object, wherein annotating the 3D model comprises:
receiving the 3D model and the 2D images, wherein a first of the 2D images is a relatively larger field of view image of the instance and a second of the 2D images is a relatively smaller field of view image of the instance;
matching signs of use that are visible in the relatively larger field of view and relatively smaller field of view images;
projecting the signs of use in the relatively larger field of view image onto the 3D model of the object;
estimating a pose of the instance in the relatively smaller field of view image using the projection of signs of use onto the 3D model and the matched signs of use in the relatively larger and relatively smaller field of view images;
computing hypothetical positions of signs of use in the relatively smaller field of view image using the estimated pose and the projection of signs of use onto the 3D model;
comparing the hypothetical positions with actual positions of the signs of use in the relatively smaller field of view image to identify improperly projected of the signs of use; and
eliminating the improperly projected of the signs of use from the projections onto the 3D model of the object.