Annotation of images based on a 3D model of objects
View Patent ↗An annotation system for providing annotations for original images is provided. In some embodiments, the annotation system accesses an annotation associated with an object of a 3D model. The annotation system also accesses and displays an original image. The annotation system renders a model image of the 3D model based on the view from which the original image was collected. When the model image contains the annotated object, the annotation system provides an indication that an annotation is associated with the object of the original image. The annotation system can provide indications of the annotation for other original images that include the annotated object irrespective of the view from which the other original images are collected.
1. A method performed by a computing system for rendering a model image of a 3D model, the method comprising:
accessing an original image along with metadata based on a view of a camera that collected the image, the metadata for determining a row number and a column number of the original image that correspond to a geospatial coordinate;
identifying an origin pixel of the original image, the origin pixel having an origin row number and an origin column number and an image height and an image width in pixels;
using the metadata to identify an origin geospatial coordinate associated with the origin pixel;
for each easting, northing, and height direction, generating a row transform factor and a column transform factor to transform geospatial coordinates of the 3D model to a clip range based on a pixel that corresponds to a geospatial coordinate that is a certain geospatial distance in that direction from the origin geospatial coordinate;
generating a perspective transform matrix based on the row transform factors and the column transform factors; and
applying the perspective transform matrix to the 3D model to generate the model image.
2. The method of claim 1 wherein for each direction, the generating of the row transform factor and the column transform factor includes:
identifying a sample row number and a sample column number relative to the origin row number and origin column number of a sample pixel for that direction corresponding to the certain geospatial distance in that direction from the origin geospatial coordinate;
setting the row transform factor based on a number of pixels per distance derived from the sample row number and the image height; and
setting the column transform factor based on a number of pixels per distance derived from the sample column number and the image width.
3. The method of claim 1 wherein the original image is a satellite image and the metadata includes rational polynomial coefficients.
4. The method of claim 1 further comprising generating a new perspective transform matrix based on a different origin and applying the new perspective transform matrix to the 3D model to generate a new model image corresponding to a different portion of the original image.
5. The method of claim 4 wherein the new model image is generated to account for perspective in the original image.
6. The method of claim 1 wherein the 3D model includes an object and the original image includes the object and further comprising:
displaying the original image; and
when a user selects the object in the original image and provides an annotation, associating the annotation with the object of the 3D model.
7. The method of claim 6 further comprising:
accessing another original image that includes the object, the other original image collected from a view that is different from a view from which the original image was collected;
displaying the other original image; and
providing an indication that an annotation is associated with the object of the other original image based on the annotation associated with the object of the 3D model.
8. The method of claim 7 further comprising determining that the original image includes the object by rendering the model image from a view from which the original image was collected so that the objects in the model image can be identified.
9. The method of claim 8 wherein objects of the 3D model are color-coded so that when a user selects a pixel in the original image, the pixel in the model image is used to identify the corresponding object.
10. The method of claim 1 wherein an annotation is associated with an object of the 3D model and further comprising:
displaying the original image; and
when the original image includes the object, providing an indication that an annotation is associated with the object of the original image.
11. The method of claim 10 further comprising displaying on the original image an annotation indicator and, when a user selects the object, providing the annotation to the user.
12. The method of claim 10 wherein objects of the 3D model are color-coded so that when a user selects a pixel in the original image, a corresponding pixel in the model image is used to identify the object based on the color coding.
13. A computing system for generating a model image of a 3D model, the computing system comprising:
one or more computer-readable storage mediums storing computer-executable instructions for controlling the computing system to:
access an original image along with metadata based on a view of a camera that collected the image, the metadata for determining a row number and a column number of the original image that correspond to a geospatial coordinate;
identify an origin pixel of the original image, the origin pixel having an origin row number and an origin column number and an image height and an image width in pixels;
identify based on the metadata an origin geospatial coordinate associated with the origin pixel;
for each easting, northing, and height direction, generate a row transform factor and a column transform factor to transform geospatial coordinates of the 3D model to a clip range based on a pixel that corresponds to a geospatial coordinate that is a certain geospatial distance in that direction from the origin geospatial coordinate;
generate a perspective transform matrix based on the row transform factors and the column transform factors; and
apply the perspective transform matrix to the 3D model to generate the model image; and
one or more processors for executing the computer-executable instructions stored in the one or more computer-readable storage mediums.
14. The computing system of claim 13 wherein for each direction, the instructions that generate the row transform factor and the column transform factor control the computing system to:
identify a sample row number and a sample column number relative to the origin row number and origin column number of a sample pixel for that direction corresponding to the certain geospatial distance in that direction from the origin geospatial coordinate;
set the row transform factor based on a number of pixels per distance derived from the sample row number and the image height; and
set the column transform factor based on a number of pixels per distance derived from the sample column number and the image width.
15. The computing system of claim 13 wherein the original image is a satellite image and the metadata includes rational polynomial coefficients.
16. The computing system of claim 13 wherein the computer-executable instructions further control the computing system to generate a new perspective transform matrix based on a different origin and apply the new perspective transform matrix to the 3D model to generate a new model image corresponding to a different portion of the original image.
17. The computing system of claim 16 wherein the new model image is generated to account for perspective in the original image.
18. The computing system of claim 13 wherein an annotation is associated with an object of the 3D model and wherein the computer-executable instructions further control the computing system to:
display the original image; and
when the original image includes the object, provide an indication that an annotation is associated with the object of the original image.
19. The computing system of claim 18 wherein the computer-executable instructions further control the computing system to display on the original image an annotation indicator and, when a user selects the object, provide the annotation to the user.
20. The computing system of claim 19 wherein objects of the 3D model are color-coded so that when a user selects a pixel in the original image, a corresponding pixel in the model image is used to identify the object based on the color coding.
21. The computing system of claim 13 wherein the 3D model includes an object and the original image includes include the object and the computer-executable instructions further control the computing system to display the original image and when a user selects the object in the original image and provides an annotation, associate the annotation with the object of the 3D model.
22. The computing system of claim 21 where the computer-executable instructions further control the computing system to:
access another original image that includes the object, the other original image collected from a view that is different from a view from which the original image was collected;
display the other original image; and
provide an indication that an annotation is associated with the object of the original image based on the annotation associated with the object of the 3D model.
23. The computing system of claim 22 wherein the computer-executable instructions further control the computing system to determine that the original image includes the object by rendering the model image from a view from which the original image was collected so that the objects in the model image can be identified.
24. The computing system of claim 23 wherein objects of the 3D model are color-coded so that when a user selects a pixel in the original image, the pixel in the model image is used to identify the corresponding object.