Repeatability predictions of interest points
The present disclosure describes approaches for evaluating interest points for localization uses based on a repeatability of the detection of the interest point in images capturing a scene that includes the interest point. The repeatability of interest points is determined by using a trained repeatability model. The repeatability model is trained by analyzing a time series of images of a scene and determining repeatability functions for each interest point in the scene. The repeatability function is determined by identifying which images in the time series of images allowed for the detection of the interest point by an interest point detection model.
1. A computer-implemented method, comprising:
receiving an identification of a location of a client device;
identifying a plurality of interest points located in the vicinity of the location of the client device;
determining a repeatability score of each of the identified interest points;
selecting a subset of interest points based on the determined repeatability score of each of the identified interest points;
generate a summary map of a vicinity of the location of the client device based on the selected subset of interest points; and
sending the summary map to the client device.
2. The method of claim 1 , further comprising:
storing, in a 3D map, information about each interest point in the selected subset of interest points.
3. The method of claim 1 , wherein determining a repeatability score for an identified interest point, comprises:
applying a repeatability model based on the identified interest point, the repeatability model trained based on a time series of images of a scene across a set time period.
4. The method of claim 3 , further comprising:
determining a current time,
wherein the repeatability score of the identified interest point is further based on the determined current time.
5. The method of claim 4 , wherein the current time is determined based on at least one of a received time from the client device, or the identified location of the client device and an internal time of a server.
6. The method of claim 3 , wherein the repeatability model is trained by:
receiving the time series of images of the scene across the set time period;
identifying, using an interest point detection model, a set of training interest points in the received time series of images;
for each training interest point in the set of training interest points, determining a repeatability function by identifying images in the time series of images in which the training interest point is detected by the interest point detection model; and
training the repeatability model using information associated with one or more training interest points from the set of training interest points and corresponding repeatability functions of the one or more training interest points.
7. The method of claim 1 , wherein the repeatability score is indicative of a likelihood that a trained interest point detection model will detect the interest point based on images captured by the client device.
8. A non-transitory computer readable storage medium configured to store instructions, the instructions that, when executed by a processor, cause the processor to:
receive an image of a scene;
identify a set of interest points from the received image of the scene;
determine a repeatability score of each interest point in the set of interest points;
select a subset of interest points based on the determined repeatability of each of the identified interest points;
search, in a map, one or more interest points from the selected subset of interest points;
receive information about one or more interest points from the selected subset of interest points; and
determine a pose of a camera that captured the image of the scene based on the received information about the one or more interest point.
9. The non-transitory computer readable storage medium of claim 8 , wherein the set of interest points are identified based on a trained interest point detection model.
10. The non-transitory computer readable storage medium of claim 8 , wherein the repeatability score for an identified interest point is determined by applying a repeatability model based on the identified interest point.
11. The non-transitory computer readable storage medium of claim 10 , wherein the repeatability model is trained based on a time series of images of a scene across a set time period.
12. The non-transitory computer readable storage medium of claim 10 , wherein searching, in a map, one or more interest points from the selected subset of interest points comprises searching the one or more interest points in a three-dimensional (3D) map.
13. The non-transitory computer readable storage medium of claim 8 , wherein determining a pose of a camera that captured the image of the scene comprises:
determining a relative pose of the camera based on the received information about the one or more interest points; and
determining an absolute pose of the camera based on a set of reference images and the determined relative pose of the camera.
14. The non-transitory computer readable storage medium of claim 8 , wherein the scene corresponds to a real-world location that maps to a virtual location in a virtual world.
15. A computer-implemented method comprising:
receiving a time series of images of a scene across a set time period;
identifying, using an interest point detection model, a set of interest points in the received time series of images;
for each interest point in the set of interest points, determining a repeatability function by identifying images in the time series of images in which the interest point is detected by the interest point detection model; and
training a repeatability model using information associated with one or more interest points from the set of interest points and corresponding repeatability functions of the one or more interest points.
16. The method of claim 15 , wherein each image in the times series of images is associated with a timestamp, and wherein the repeatability function is further based on the timestamp associated with each image in which the interest point is detected by the interest point detection model.
17. The method of claim 15 , wherein the repeatability function indicates, for each time period of a set of time period, whether the interest point was detected, by the interest point detection model, in an image corresponding to the time period.
18. The method of claim 15 , wherein the repeatability function indicates, for each time period of a set of time period, a percentage of images corresponding to the time period in which the interest point was detected by the interest point detection model.
19. The method of claim 15 , wherein the repeatability model is trained to predict a likelihood that a target interest point will be detected by the interest point detection model from an image captured within a given time interval.
20. The method of claim 15 , wherein the repeatability model is trained to predict a time series of likelihoods that a target interest point will be detected by the interest point detection model from an image of the scene where the interest point is located.