Simulated consistency check for points of interest on three-dimensional maps
The present disclosure describes approaches to camera re-localization that improve the accuracy of re-localization determinations by performing simulated consistency checks for three-dimensional maps. Client devices associated with users of a location-based application transmit image scans to a game server, which divides the received scan data into mapping sets used to generate 3D maps of environments and validation sets used to test the accuracy of the maps. To perform the testing, the game server identifies query scans in the validation set having GPS coordinates within a threshold distance of the mapped location and uses the 3D map of the environment to generate a pose estimate for each frame. The results of the localization queries are analyzed by comparing differences between the localization pose estimates and differences between the poses of independent pairs of frames in the query scan to evaluate the accuracy of the 3D map.
1. A computer-implemented method comprising:
selecting a three-dimensional (3D) map of an environment for testing, the 3D map including locations of one or more points of interest within the environment and associated with a location-based application;
identifying a query scan comprising a plurality of frames captured by a camera assembly on a client device positioned at a point of interest within the environment;
selecting independent pairs of frames from the query scan;
for each frame of the selected pairs of frames, generating, by a localization model and using the 3D map, a pose estimate representing a prediction of a 3D position and orientation of the client device;
for each pair of frames, comparing the pose estimates with relative position information determined based on sensor data from the client device; and
generating a localizability score for the point of interest based on differences between the pose estimates and the relative position information for each pair of frames.
2. The computer-implemented method of claim 1 , wherein generating the localizability score comprises:
generating a confidence score of the pose estimate, the confidence score representing a probability that the pose estimate is correct;
calculating a scan quality score for the query scan, the scan quality score indicating a level of localization and consistency of the query scan; and
combining the confidence score and the scan quality score to generate the localizability score.
3. The computer-implemented method of claim 1 , wherein the localizability score for a point of interest represents a probability that the localization model will generate a pose estimate of a client device to at least a threshold level of accuracy within a specified time period.
4. The computer-implemented method of claim 1 , further comprising selecting a location for an augmented reality event associated with the location-based application based on localizability scores for points of interest within the environment represented by the 3D map.
5. The computer-implemented method of claim 1 , wherein the 3D map of the environment is generated using a plurality of sets of image data and sensor data captured by camera assemblies on a plurality of client devices.
6. The computer-implemented method of claim 1 , further comprising:
storing localizability scores for a plurality of points of interest within environments represented by 3D maps;
receiving, from a second location-based application, GPS coordinates for a client device associated with the second location-based application;
selecting a 3D map associated with the received GPS coordinates; and
selecting one or more points of interest within the 3D map using the stored localizability scores.
7. The computer-implemented method of claim 1 , wherein the sensor data comprises readings from one or more of a movement sensor, an accelerometer, a gyroscope, an inertial measurement unit, a barometer, a positioning system, a thermometer, a light sensor, and a depth sensor on the client device.
8. A non-transitory computer-readable storage medium comprising instructions executable by a computing device, the instructions, when executed, causing the computing device to perform operations including:
selecting a three-dimensional (3D) map of an environment for testing, the 3D map including locations of one or more points of interest within the environment and associated with a location-based application;
identifying a query scan comprising a plurality of frames captured by a camera assembly on a client device positioned at a point of interest within the environment;
selecting independent pairs of frames from the query scan;
for each frame of the selected pairs of frames, generating, by a localization model and using the 3D map, a pose estimate representing a prediction of a 3D position and orientation of the client device;
for each pair of frames, comparing the pose estimates with relative position information determined based on sensor data from the client device; and
generating a localizability score for the point of interest based on differences between the pose estimates and the relative position information for each pair of frames.
9. The non-transitory computer-readable storage medium of claim 8 , wherein generating the localizability score comprises:
generating a confidence score of the pose estimate, the confidence score representing a probability that the pose estimate is correct;
calculating a scan quality score for the query scan, the scan quality score indicating a level of localization and consistency of the query scan; and
combining the confidence score and the scan quality score to generate the localizability score.
10. The non-transitory computer-readable storage medium of claim 8 , wherein the localizability score for a point of interest represents a probability that the localization model will generate a pose estimate of a client device to at least a threshold level of accuracy within a specified time period.
11. The non-transitory computer-readable storage medium of claim 8 , wherein the operations further comprise selecting a location for an augmented reality event associated with the location-based application based on localizability scores for points of interest within the environment represented by the 3D map.
12. The non-transitory computer-readable storage medium of claim 8 , wherein the 3D map of the environment is generated using a plurality of sets of image data and sensor data captured by camera assemblies on a plurality of client devices.
13. The non-transitory computer-readable storage medium of claim 8 , wherein the operations further comprise:
storing localizability scores for a plurality of points of interest within environments represented by 3D maps;
receiving, from a second location-based application, GPS coordinates for a client device associated with the second location-based application;
selecting a 3D map associated with the received GPS coordinates; and
selecting one or more points of interest within the 3D map using the stored localizability scores.
14. The non-transitory computer-readable storage medium of claim 8 , wherein the sensor data comprises readings from one or more of a movement sensor, an accelerometer, a gyroscope, an inertial measurement unit, a barometer, a positioning system, a thermometer, a light sensor, and a depth sensor on the client device.
15. A computer-implemented method comprising:
selecting a three-dimensional (3D) model of an environment for testing, the 3D model including locations of one or more points of interest within the environment;
generating, for each point of interest within the 3D model of the environment, an accuracy score representing a likelihood of accurately predicting a 3D position and orientation of a client device using the 3D model; and
selecting a location for an augmented reality event based on the accuracy scores for the points of interest within the environment.
16. The computer-implemented method of claim 15 , wherein generating an accuracy score comprises:
for each of a plurality of pairs of camera frames contained in a query scan of frames captured at the point of interest by a camera assembly on a client device, comparing client device sensor data associated with the frames with pose estimates for the frames generated by a localization model using the 3D model; and
generating the accuracy score based on differences between the sensor data and the pose estimates.
17. The computer-implemented method of claim 16 , further comprising generating a localizability score for each point of interest, wherein generating the localizability score comprises:
generating a confidence score of the pose estimate for the point of interest, the confidence score representing a probability that the pose estimate is correct;
calculating a scan quality score for the query scan, the scan quality score indicating a level of localization and consistency of the query scan; and
combining the confidence score and the scan quality score to generate the localizability score.
18. The computer-implemented method of claim 16 , wherein the sensor data comprises readings from one or more of a movement sensor, an accelerometer, a gyroscope, an inertial measurement unit, a barometer, a positioning system, a thermometer, a light sensor, and a depth sensor on the client device.
19. The computer-implemented method of claim 15 , wherein the 3D model of the environment is generated using a plurality of sets of image data and sensor data captured by camera assemblies on a plurality of client devices.
20. The computer-implemented method of claim 15 , wherein selecting a location for an augmented reality event based on the accuracy score comprises:
receiving, from a location-based application, GPS coordinates for a client device associated with the location-based application; and
identifying a point of interest within a threshold distance of the GPS coordinates and having an accuracy score exceeding a score threshold.