Localisation and mapping
View Patent ↗A method generates a three-dimensional map of a region from successive images of that region captured from different camera poses. The method captures successive images of the region, detects a gravitational vertical direction in respect of each captured image, detects feature points within the captured images and designates a subset of the captured images as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that image. The method also includes, for a captured image (i) deriving a camera pose from detected feature points in the image; (ii) rotating the gravitational vertical direction to the coordinates of a reference keyframe using the camera poses derived for that image and the reference keyframe; and (iii) comparing the rotated direction with the actual gravitational vertical direction for the reference keyframe to detect a quality measure of that image.
1. A method of generating a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with that three dimensional position, the method comprising:
capturing successive images of the region;
detecting, by a processor, a gravitational vertical direction in respect of each captured image;
detecting, by the processor, feature points within the captured images;
designating, by the processor, a subset of the captured images as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that keyframe;
in respect of a given image captured at a relocalisation operation:
(i) deriving, by the processor, a camera pose for the given image from detected feature points in the given image;
(ii) rotating, by the processor, the gravitational vertical direction in respect of the given image to the coordinates of a reference keyframe using the camera poses derived for the given image and the reference keyframe; and
(iii) comparing, by the processor, the rotated gravitational vertical direction with an actual gravitational vertical direction associated with the reference keyframe so as to detect a quality measure of the given image.
2. A method according to claim 1 , in which the reference keyframe is an earliest-captured one of the set of keyframes.
3. A method according to claim 1 , comprising:
performing tests to detect whether or not to designate the given image as a keyframe, outcomes of the tests varying according to a detected quality measure for that candidate keyframe.
4. A method according to claim 1 , in which the actual gravitational vertical direction associated with the reference keyframe depends upon a gravity direction associated with two or more keyframes.
5. A method according to claim 1 , in which, if the rotated gravitational vertical direction with the actual gravitational vertical direction associated with the reference keyframe differ by more than a threshold difference, the deriving step is repeated in respect of a different set of landmark points.
6. A non-transitory computer readable medium storing computer instructions thereon, the instructions, when executed by a processor, cause the processor to carry out a method of generating a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with that three dimensional position, the method comprising:
capturing successive images of the region;
detecting a gravitational vertical direction in respect of each captured image;
detecting feature points within the captured images;
designating a subset of the captured images as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that keyframe;
in respect of a given image captured at a relocalisation operation:
(i) deriving a camera pose for the given image from detected feature points in the given image;
(ii) rotating the gravitational vertical direction in respect of the given image to the coordinates of a reference keyframe using the camera poses derived for the given image and the reference keyframe; and
(iii) comparing the rotated gravitational vertical direction with an actual gravitational vertical direction associated with the reference keyframe so as to detect a quality measure of the given image.
7. Image processing apparatus configured to generate a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with that three dimensional position, a subset of the captured images being designated as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that keyframe, wherein the apparatus comprises:
a camera configured to capture successive images of the region;
a gravity detector configured to detect a gravitational vertical direction in respect of each captured image;
a relocalisation controller configured, in respect of a given image captured at a relocalisation operation, to:
(i) derive a camera pose for the given image from detected landmark points in that keyframe;
(ii) rotate the gravitational vertical direction in respect of the given image to the coordinates of a reference keyframe using the camera poses derived for the given image and the reference keyframe; and
(iii) compare the rotated gravitational vertical direction with an actual gravitational vertical direction associated with the reference keyframe so as to detect a quality measure of the given image.
8. Computer games apparatus comprising:
an image processing apparatus configured to generate a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with that three dimensional position, a subset of the captured images being designated as a set of keyframes each having respective sets of image position data representing image positions of landmark points detected as feature points in that keyframe, wherein the image processing apparatus comprises:
a camera configured to capture successive images of the region;
a gravity detector configured to detect a gravitational vertical direction in respect of each captured image;
a relocalisation controller configured, in respect of a given image captured at a relocalisation operation, to:
(i) derive a camera pose for the given image from detected landmark points in that keyframe;
(ii) rotate the gravitational vertical direction in respect of the given image to the coordinates of a reference keyframe using the camera poses derived for the given image and the reference keyframe; and
(iii) compare the rotated gravitational vertical direction with an actual gravitational vertical direction associated with the reference keyframe so as to detect a quality measure of the given image; and
an image renderer configured to render an image for display by combining a captured image with one or more computer-generated image portions generated with respect to map positions associated with feature points of the captured image.