Annotation of 3D models with signs of use visible in 2D images
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for annotation of 3D models with signs of use that are visible in 2D images. In one aspect, methods are performed by data processing apparatus. The methods can include projecting signs of use in a relatively larger field of view image of an instance of an object onto a 3D model of the object based on a pose of the instance in the relatively larger field of view image, and estimating a relative pose of the instance of the object in a relatively smaller field of view image based on matches between the signs of use in the relatively larger field of view image and the same signs of use in the relatively smaller field of view image.
1 . A method performed by data processing apparatus, the method comprising:
estimating a relative pose of an instance of an object in a relatively smaller field of view image based on matched signs of use of the object that appear in both a relatively larger field of view image and the relatively smaller field of view image;
identifying, using the relative pose, matched signs of use that satisfy a threshold likelihood that the matched signs of use are correct matches; and
projecting the matched signs of use that satisfy the threshold likelihood onto a 3D model of the object.
2 . The method of claim 1 further comprising:
determining a dominant color of the instance of the object in the relatively smaller field of view image;
identifying regions in the relatively larger field of view image, in the relatively smaller field of view image, or in both the relatively larger field of view image and the relatively smaller field of view image that deviate from the dominant color; and
matching the identified regions to match the signs of use in the relatively larger field of view image and the signs of use in the relatively smaller field of view image.
3 . The method of claim 1 , further comprising:
determining a deviation from ideality in the instance of the object in the relatively smaller field of view image; and
matching the deviation in the relatively smaller field of view image to the relatively larger field of view image to match the signs of use in the relatively larger field of view image and the signs of use in the relatively smaller field of view image.
4 . The method of claim 1 , further comprising:
computing hypothetical positions of signs of use in the relatively smaller field of view image using the estimated pose;
comparing the hypothetical positions with actual positions of the signs of use in the relatively smaller field of view image; and
identifying, based on the comparison, a subset of the signs of use that are improperly projected onto the 3D model.
5 . The method of claim 1 , wherein projecting the matched signs of use onto the 3D model comprises determining a pose of the relatively larger field of view image.
6 . The method of claim 1 , further comprising:
computing hypothetical positions of the signs of use in the relatively smaller field of view image using the estimated pose and the projection of signs of use onto the 3D model;
comparing the hypothetical positions with actual positions of the signs of use in the relatively smaller field of view image; and
identifying, based on the comparison, a subset of the signs of use that are improperly projected onto the 3D model.
7 . The method of claim 6 , wherein the improperly projected signs of use are identified based on a positional deviation between the hypothetical positions and the actual positions of the signs of use in the relatively smaller field of view image.
8 . The method of claim 6 , wherein the method comprises:
filtering the subset of the signs of use that are improperly projected onto the 3D model from the matches to establish a proper subset of the matches; and
re-estimating the relative pose of the instance of the object in the relatively smaller field of view image based on subset of the matches; and
projecting the proper subset of the matches onto the 3D model of the object using the re-estimated relative pose.
9 . The method of claim 1 , wherein projecting the matched signs of use that satisfy the threshold likelihood onto the 3D model of the object comprises:
identifying a region of the relatively smaller field of view image that includes a first sign of use that satisfies the threshold likelihood;
matching the first sign of use to a first of the signs of use in the relatively larger field of view image; and
projecting the first of the signs of use in the relatively larger field of view image onto the 3D model of the object.
10 . The method of claim 1 , further comprising deforming the 3D model using projected matched signs of use.
11 . At least one computer-readable storage medium encoded with executable instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
estimating a relative pose of an instance of an object in a relatively smaller field of view image based on matched signs of use of the object that appear in both a relatively larger field of view image and the relatively smaller field of view image;
identifying, using the relative pose, matched signs of use that satisfy a threshold likelihood that the matched signs of use are correct matches; and
projecting the matched signs of use that satisfy the threshold likelihood onto a 3D model of the object.
12 . The computer-readable storage medium of claim 11 , wherein the operations further comprise:
determining a dominant color of the instance of the object in the relatively smaller field of view image;
identifying regions in the relatively larger field of view image, in the relatively smaller field of view image, or in both the relatively larger field of view image and the relatively smaller field of view image that deviate from the dominant color; and
matching the identified regions to match the signs of use in the relatively larger field of view image and the signs of use in the relatively smaller field of view image.
13 . The computer-readable storage medium of claim 12 , wherein the operations further comprise:
determining a deviation from ideality in the instance of the object in the relatively smaller field of view image; and
matching the deviation in the relatively smaller field of view image to the relatively larger field of view image to match the signs of use in the relatively larger field of view image and the signs of use in the relatively smaller field of view image.
14 . The computer-readable storage medium of claim 11 , wherein the operations further comprise:
computing hypothetical positions of signs of use in the relatively smaller field of view image using the estimated pose;
comparing the hypothetical positions with actual positions of the signs of use in the relatively smaller field of view image; and
identifying, based on the comparison, a subset of the signs of use that are improperly projected onto the 3D model.
15 . The computer-readable storage medium of claim 11 , wherein projecting the matched signs of use onto the 3D model comprises determining a pose of the relatively larger field of view image.
16 . The computer-readable storage medium of claim 11 , further comprising:
computing hypothetical positions of the signs of use in the relatively smaller field of view image using the estimated pose and the projection of signs of use onto the 3D model;
comparing the hypothetical positions with actual positions of the signs of use in the relatively smaller field of view image; and identifying, based on the comparison, a subset of the signs of use that are improperly projected onto the 3D model.
17 . The computer-readable storage medium of claim 16 , wherein the improperly projected signs of use are identified based on a positional deviation between the hypothetical positions and the actual positions of the signs of use in the relatively smaller field of view image.
18 . A method performed by a data processing apparatus, the method comprising:
generating a 3D model of an object;
projecting first signs of use onto the 3D model of the object, wherein the first signs of use comprise signs of use that are visible in a relatively larger field of view image of an instance of the object and a relatively smaller field of view image of the instance of object; and
eliminating improperly projected signs of use from the projections onto the 3D model of the object, wherein the improperly projected signs of use comprise signs of use with actual positions in the relatively smaller field of view image of the instance of object that differ from hypothetical positions of the signs of use in the relatively smaller field of view image of the instance of object, the hypothetical positions computed using an estimated pose of the instance of the object in the relatively smaller field of view image of the instance of object.
19 . The method of claim 18 , wherein projecting the first signs of use onto the 3D model comprises:
identifying a region of the relatively smaller field of view image that includes a sign of use;
matching the sign of use to a corresponding sign of use in the relatively larger field of view image; and
projecting the sign of use in the relatively larger field of view image onto the 3D model of the object.
20 . The method of claim 18 , further comprising deforming the 3D model using projected matched signs of use.