Device and method with camera pose estimation
Disclosed are a device and method with pose estimation. A method of operating an electronic device includes: obtaining monocular images captured at different respective points in time by a same camera; estimating, based on a neural network, a position change of the camera corresponding to the time between the points in time at which the images were captured; and estimating a position and/or orientation of the camera based on the estimated position change of the camera and based on the images.
1 . An electronic device, comprising:
one or more processors; and
a memory storing instructions configured to, when executed by the one or more processors, cause the one or more processors to:
obtain two single images captured at two respective points in time by a same camera,
estimate, by a simultaneous localization and mapping (SLAM) neural network, a position change of the camera that occurred during the time interval between the two points, wherein the estimating is based on the two single images, and wherein the position change is estimated as part of the SLAM neural network inferring features of the two single images, and
based on the estimated position change of the camera, adjust a position and/or orientation of the camera determined based on the inferred features of the two single images.
2 . The electronic device of claim 1 , wherein the instructions are further configured to cause the one or more processors to estimate the position and/or orientation of the camera by performing a simultaneous localization and mapping (SLAM) operation based on the inferred features, wherein the SLAM operation estimates the camera position and/or orientation based on feature point matching between feature points in the inferred features of the two images, and wherein the SLAM operation is not an operation of the SLAM network.
3 . The electronic device of claim 1 , wherein the instructions are further configured to cause the one or more processors to adjust the position and/or orientation of the camera by adjusting a scale of the estimated camera position and/or orientation according to the estimated position change of the camera.
4 . The electronic device of claim 3 , wherein
the initial estimation result for the pose variation comprises pose variations of the camera for time intervals between a current image and respective previous images, wherein the current image is one of the two images and the other of the two images is one of the previous images, and
the estimated position change of the camera comprises estimated position changes of the camera between the current image and the respective previous images,
wherein the instructions are further configured to cause the one or more processors to estimate the position and/or orientation of the camera by adjusting the scale of the initial estimation result so that an error between the pose variations and the estimated position changes of the camera is minimized.
5 . The electronic device of claim 1 , wherein adjusting the position and/or orientation of the camera based on the position change of the camera reduces scale ambiguity in a SLAM operation used to derive the position and/or orientation of the camera prior to the adjusting thereof.
6 . The electronic device of claim 5 , wherein the SLAM operation comprises a monocular visual SLAM algorithm configured for processing monocular images.
7 . The electronic device of claim 1 , wherein
the SLAM neural network is trained based on reference images collected by a reference device and based on reference position changes of a reference camera for respective time intervals between times at which the reference images were captured,
wherein the reference device comprises the reference camera, and a reference position sensor and/or a reference inertial sensor that senses positions upon which the reference position changes are based.
8 . The electronic device of claim 7 , wherein
the reference images are captured at different respective reference points in time by the reference camera, and
the reference position changes are determined by values sensed by the reference position sensor and/or the reference inertial sensor at the respective reference points in time.
9 . The electronic device of claim 1 , wherein
the electronic device is, or is comprised in: an autonomous vehicle, a vehicle equipped with a driving assistance system, a vehicle supporting advanced driver-assistance systems (ADAS), a smartphone equipped with a camera, a wearable device, or a robot.
10 . A method of operating an electronic device, the method comprising:
obtaining monocular images captured at different respective points in time by a same camera;
estimating, based on a neural network, a position change of the camera corresponding to the time between the points in time at which the images were captured, wherein the estimating is based on the monocular images of the camera and is not based on images from any other camera; and
adjusting a position and/or orientation of the camera based on the estimated position change of the camera and based on the images.
11 . The method of claim 10 , wherein
the adjusting of the position and/or orientation of the camera comprises performing a simultaneous localization and mapping (SLAM) operation based on feature point matching between the images to derive the position and/or orientation of the camera which is adjusted based on the estimated position change of the camera.
12 . The method of claim 11 , wherein
the initial estimation result for the pose variation comprises pose variations of the camera for time intervals between a current image and respective previous images, wherein the current image is one of the two images and the other of the two images is one of the previous images, and
the estimated position change of the camera comprises estimated position changes of the camera between the current image and the respective previous images,
wherein the estimating the position and/or orientation of the camera is performed by adjusting the scale of the initial estimation result so that an error between the pose variations and the estimated position changes of the camera is minimized.
13 . The method of claim 10 , wherein
the adjusting of the position and/or orientation of the camera comprises adjusting a scale of the position and/or orientation of the camera according to the position change of the camera.
14 . The method of claim 10 , wherein
the adjusting of the position and/or orientation of the camera reduces scale ambiguity in a SLAM operation that derives the position and/or orientation of the camera, prior to the adjusting thereof, based on feature point matching between the images.
15 . The method of claim 14 , wherein
the SLAM operation comprises a monocular visual SLAM operation that derives the position and/or orientation of the camera based on images of only the camera.
16 . The method of claim 10 , wherein
the neural network is trained based on reference images collected by a reference device and based on reference position changes of a reference camera for respective time intervals between times at which the reference images are captured,
wherein the reference device comprises the reference camera and a reference position sensor and/or a reference inertial sensor that senses the positions upon which the reference position changes are based.
17 . The method of claim 16 , wherein
the reference images are captured at different respective reference points in time by the reference camera, and
the reference position changes are determined by values sensed by the reference position sensor and/or the reference inertial sensor at the respective reference points in time.
18 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 10 .
19 . The method of claim 10 , wherein the position change comprises a distance.
20 . A method comprising:
inputting a first image captured by a camera at a first time and a second image captured by the camera at a second time to a simultaneous localization and mapping (SLAM) network, wherein the SLAM network, in inferring a first feature map and a second feature map from the first and second images, also outputs a camera-move distance inferred from the first and second images, the camera-move distance corresponding to movement of the camera from the first time to the second time;
matching features between the first and second feature maps to estimate an initial camera pose of the camera; and
adjusting a scale of the estimated camera pose based on the camera-move distance.