Processing images for extracting information about known objects
A computer-implemented method of processing images for extracting information about known objects comprises the steps of receiving an image containing a view of a known object at a scale dependent on an object distance of the known object from an image capture location of the image; determining, from a world model representing one or more known objects in the vicinity of the image capture location, an object location of the known object, the object location and the image capture location defined in a world frame of reference; and based on the image capture location and the object location in the world frame of reference, applying image scaling to the image, to extract a rescaled image containing a rescaled view of the known object at a scale that is substantially independent of the object distance from the image capture location.
1 . A computer-implemented method of processing images for extracting information about known objects, the method comprising:
receiving an image containing a view of a known object at a scale dependent on an object distance of the known object from an image capture location of the image;
determining, from a world model representing one or more known objects at a distance from the image capture location, an object location of the known object, the object location and the image capture location defined in a world frame of reference; and
based on the image capture location and the object location in the world frame of reference, applying image scaling to the image, to extract a rescaled image containing a rescaled view of the known object at a scale that is independent of the object distance from the image capture location,
wherein the world model comprises an object model for modelling the known object and an object projection is computed by projecting the object model from the object location into an image plane of the image based on the image capture location, the object projection used to determine the image scaling applied to the image.
2 . The method of claim 1 , wherein the image is received with ego localization data defining the image capture location in the world frame of reference, the ego localization data computed via localization.
3 . The method of claim 2 , wherein:
the world model comprises a predetermined map of one or more known static objects, the known object being a known static object and the world frame of reference being a frame of reference of the predetermined map;
the image is one of a time sequence of images; and
the ego localization data has been computed in the world frame of reference by applying vision-based localization to the time sequence of images, or
the ego localization data has been computed in the world frame of reference by applying non-vision-based localization to at least one other type of sensor data associated with the image, or
the ego localization data has been computed using a combination of vision-based and non-vision based localization.
4 . The method of claim 1 , comprising inputting the rescaled image to an image recognition component wherein:
the image recognition component processes the rescaled image to extract information from the rescaled view of the known object; or
the image recognition component processes the rescaled image to extract information from the rescaled view of the known object, and wherein the image recognition component has been configured to operate at said scale.
5 . The method of claim 4 , wherein:
the image recognition component is configured to classify the rescaled view of the known object in relation to a set of predetermined object states, or
the image recognition component is configured to classify the rescaled view of the known object in relation to a set of predetermined object states, and wherein the set of predetermined object states is a set of predetermined signalling states.
6 . The method of claim 1 , applied to multiple images containing respective views of the known object at different scales,
in order to extract respective rescaled images, containing respective rescaled views of the known object at the same scale, or
in order to extract respective rescaled images, containing respective rescaled views of the known object at the same scale, wherein the multiple images form part of a time sequence of images captured over a sequence of image capture locations.
7 . The method of claim 5 , wherein:
the set of predetermined object states is a set of predetermined signalling states; and
the method is applied in order to extract a scenario for running in a simulator, the scenario encoding a state change of the known object.
8 . The method of claim 6 , comprising using the multiple rescaled images to train an image recognition component to extract information about the known object when viewed at said scale.
9 . The method of claim 1 , comprising computing, in an image plane of the image, based on the image capture location and the object location as determined in the world frame of reference, a crop region containing the view of the known object, wherein the rescaled image is a cropped and rescaled image extracted from a portion of the image within the crop region.
10 . The method of claim 1 , wherein the rescaled image is displayed on an annotation interface, the method comprising generating manual annotation data for annotating a location of the known object in the rescaled image in response to manual annotation inputs at the annotation interface.
11 . The method of claim 10 , applied to multiple images containing respective views of the known object at different scales, in order to extract respective rescaled images, containing respective rescaled views of the known object at the same scale,
wherein:
the multiple images form part of a time sequence of images captured over a sequence of image capture locations, and the method comprises calculating interpolated or extrapolated annotation data for annotating a location of the object in at least one other image of the time sequence of images via interpolation or extrapolation of the manual annotation data; or
the multiple images form part of a time sequence of images captured over a sequence of image capture locations, and the method comprises calculating interpolated or extrapolated annotation data for annotating a location of the object in at least one other image of the time sequence of images via interpolation or extrapolation of the manual annotation data; and
wherein the image capture location and the object location are used to annotate the image with an assumed location of the object in an image plane, the manual annotation inputs for correcting the assumed location of the object.
12 . The method of claim 11 , comprising computing, in an image plane of the image, based on the image capture location and the object location as determined in the world frame of reference, a crop region containing the view of the known object, wherein the rescaled image is cropped and extracted from a portion of the image within the crop region,
wherein the assumed location is a predefined location within the crop region.
13 . The method of claim 10 , comprising updating ego localization data in response to the manual annotation inputs, the updated localization data for determining an updated image capture location of the image consistent with the manual annotation data,
wherein the image is received with ego localization data defining the image capture location in the world frame of reference, the ego localization data computed via localization.
14 . The method of claim 1 , comprising computing, in an image plane of the image, based on the image capture location and the object location as determined in the world frame of reference, a crop region containing the view of the known object, wherein the rescaled image is a cropped and rescaled image extracted from a portion of the image within the crop region,
wherein the object projection is used to determine the crop region.
15 . The method of claim 1 , wherein the world frame of reference has three spatial dimensions and the object model is a 3D object model.
16 . The method of claim 1 , wherein the known object is a moving object.
17 . A computer system comprising:
one or more computers configured to:
receive an image containing a view of a known object at a scale dependent on an object distance of the known object from an image capture location of the image;
determine, from a world model representing one or more known objects at a distance from the image capture location, an object location of the known object, the object location and the image capture location defined in a world frame of reference; and
based on the image capture location and the object location in the world frame of reference, apply image scaling to the image, to extract a rescaled image containing a rescaled view of the known object at a scale that is independent of the object distance from the image capture location,
wherein the world model comprises an object model for modelling the known object and an object projection is computed by projecting the object model from the object location into an image plane of the image based on the image capture location, the object projection used to determine the image scaling applied to the image.
18 . A non-transitory media embodying computer-readable instructions configured, upon execution on one or more processors, to execute a method including:
receiving an image containing a view of a known object at a scale dependent on an object distance of the known object from an image capture location of the image;
determining, from a world model representing one or more known objects at a distance from the image capture location, an object location of the known object, the object location and the image capture location defined in a world frame of reference; and
based on the image capture location and the object location in the world frame of reference, applying image scaling to the image, to extract a rescaled image containing a rescaled view of the known object at a scale that is independent of the object distance from the image capture location,
wherein the world model comprises an object model for modelling the known object and an object projection is computed by projecting the object model from the object location into an image plane of the image based on the image capture location, the object projection used to determine the image scaling applied to the image.
19 . The computer system of claim 17 , embodied in a mobile robot.