Devices and methods for visual localization
A computing device is provided for supporting a mobile device to perform a visual localization in a target environment. The computing device is configured to obtain and modify one or more 3D models of the target environment to generate a set of 3D models. The computing device is further configured to determine a set of training samples based on the set of 3D models. Each training sample comprises an image derived from the set of the 3D models and corresponding pose information. The computing device is further configured to train a neural network model based on the set of training samples, so that the trained neural network model is configured to predict pose information for an image captured in the target environment and to output a confidence value associated with the predicted pose information.
1 . A computing device for supporting a mobile device to perform a visual localization in a target environment, the computing device comprising:
a memory configured to store instructions; and
a processor coupled to the memory and configured to execute the instructions to cause the computing device to:
obtain one or more three-dimensional (3D) models of the target environment;
dynamically generate a set of 3D models of the target environment by modifying the obtained one or more 3D models of the target environment according to changes in the target environment;
determine a set of training samples, wherein each training sample comprises an image derived from the set of 3D models and corresponding pose information in the target environment; and
train a model based on the set of training samples, wherein the trained model is configured to predict pose information for an image obtained in the target environment and output a confidence value associated with the predicted pose information.
2 . The computing device according to claim 1 , wherein the one or more 3D models comprise a plurality of different 3D models of the target environment.
3 . The computing device according to claim 1 , wherein the image of the target environment comprises at least one of:
a picture of the target environment;
a depth map of the target environment; and
a scan of the target environment.
4 . The computing device according to claim 1 , wherein the processor is further configured to execute the instructions to cause the computing device to:
model an artificial mobile device including one or more visual sensors; and
determine the set of training samples further based on the artificial mobile device.
5 . The computing device according to claim 1 , wherein the processor is further configured to execute the instructions to cause the computing device to:
compute a set of trajectories, wherein each trajectory models a movement of the mobile device in the target environment; and
determine the set of training samples further based on the set of trajectories.
6 . The computing device according to claim 1 , wherein modifying the one or more 3D models of the target environment comprises at least one of:
augmenting the 3D model of the target environment with one or more artificial sources of illumination; and
adding to the 3D model of the target environment one or more static and/or dynamic objects.
7 . The computing device according to claim 1 , wherein modifying the one or more 3D models of the target environment comprises applying a domain translation operation on the one or more 3D models of the target environment.
8 . The computing device according to claim 1 , wherein the processor is further configured to execute the instructions to cause the computing device to:
apply a domain translation operation or other data augmentation operation on one or more training samples of the set of training samples before training the model based on the set of training samples.
9 . The computing device according to claim 1 , wherein the processor is further configured to execute the instructions to cause the computing device to:
provide the trained model to the mobile device;
receive feedback information from the mobile device, wherein the feedback information comprises an image obtained in the target environment and corresponding pose information; and
retrain the trained model based on the feedback information.
10 . The computing device according claim 9 , wherein the processor is further configured to execute the instructions to cause the computing device to:
provide the trained model to one or more further mobile devices;
receive feedback information from each of the one or more further mobile devices; and
retrain the trained model based on all the received feedback information.
11 . The computing device according to claim 10 , wherein the set of training samples comprises:
a first subset of training samples for training the model to predict pose information for images obtained in the target environment; and
a second subset of training samples, disjoint to the first subset of training samples, for training the model to provide confidence values for predictions of pose information.
12 . The computing device according to claim 1 , wherein the computing device is a mobile device and is configured to train and/or retrain the model in collaboration with one or more further mobile devices.
13 . The computing device according to claim 1 , wherein the processor is further configured to execute the instructions to cause the computing device to request the one or more 3D models of the target environment from one or more mapping platforms for producing 3D models.
14 . A mobile device for performing a visual localization in a target environment, the mobile device comprising:
a memory configured to store instructions; and
a processor coupled to the memory and configured to execute the instructions to cause the mobile device to:
receive a trained model from a computing device, wherein the trained model is generated based on a set of training samples according to a set of 3D models that are dynamically generated according to changes in the target environment;
capture an image in the target environment;
input the image into the trained model, wherein the trained model is configured to predict pose information corresponding to the image and to output a confidence value associated with the predicted pose information; and
in response to the confidence value being below a threshold, provide feedback information to the computing device, the feedback information comprising the image and the predicted pose information.
15 . The mobile device according to claim 14 , wherein the image of the target environment comprises at least one of:
a picture of the target environment;
a depth map of the target environment; and
a scan of the target environment.
16 . A method for supporting a mobile device to perform a visual localization in a target environment, wherein the method is applied to a computing device and comprises:
obtaining one or more three-dimensional (3D) models of the target environment;
generating a set of 3D models of the target environment by modifying the obtained one or more 3D models of the target environment;
determining a set of training samples, wherein each training sample comprises an image derived from the set of 3D models and corresponding pose information in the target environment; and
training a model based on the set of training samples, wherein the trained model is configured to predict pose information for an image obtained in the target environment and output a confidence value associated with the predicted pose information.
17 . A non-transitory computer readable medium comprising a program code which, upon execution by a computer, cause the computer to perform the method according to claim 16 .
18 . The method according to claim 16 , wherein the one or more 3D models comprise a plurality of different 3D models of the target environment.
19 . A method for a mobile device performing a visual localization in a target environment, wherein the method comprises:
receiving a trained model from a computing device, wherein the trained model is generated based on a set of training samples according to a set of three-dimensional (3D) models that are dynamically generated according to changes in the target environment;
capturing an image in the target environment;
inputting the image into the trained model, wherein the trained model is configured to predict pose information corresponding to the image and to output a confidence value associated with the predicted pose information; and
in response to the confidence value being below a threshold, providing feedback information to the computing device, the feedback information comprising the image and the predicted pose information.
20 . The method according to claim 19 , wherein the image of the target environment comprises at least one of:
a picture of the target environment;
a depth map of the target environment; and
a scan of the target environment.