System and method for improving localization and object tracking
In one embodiment, a computing system is configured to, during a first tracking session, detect first landmarks in a first image of the environment surrounding a user, and determine a first location of the user by comparing detected first landmarks to a landmark database. During a second tracking session, the computing system captures motion data and estimates a second location of the user based on the motion data and first user location. Based on the motion data and first user location, the computing system detects landmarks in a second image at a second location. The system accesses expected landmarks from the landmark database visible at the second location and determines the estimated second location of the user is inaccurate by comparing the expected landmarks with the second landmarks. The computing system re-localizes the user by comparing the landmarks in the landmark database and third landmarks in a third image.
1. A method comprising, by a computing system:
during a first tracking session:
detecting first landmarks in a first image of an environment surrounding a user; and
determining a first location of the user by comparing the first landmarks to landmarks in a landmark database;
during a second tracking session:
capturing motion data using a motion sensor; and
estimating, without using the landmark database, a second location of the user based on the motion data and the first location;
detecting second landmarks in a second image captured from a viewpoint of the user at the second location;
accessing, from the landmark database, expected landmarks visible from the viewpoint of the user at the second location;
determining, based on a scoring criteria, that the estimated second location of the user is inaccurate by comparing the expected landmarks with the second landmarks; and
re-localizing the user by comparing the landmarks in the landmark database and third landmarks detected in a third image.
2. The method of claim 1 , wherein detecting second landmarks in a second image captured from a viewpoint of the user at the second location comprises:
for each of the one or more expected landmarks visible from the viewpoint of the user at the second location, searching for a corresponding landmark within the one or more detected second landmarks in the second image.
3. The method of claim 1 , wherein the scoring criteria are based on semantic information associated with the expected landmarks.
4. The method of claim 1 , wherein re-localizing the user is further based on a determination that the scoring criteria exceeds a predetermined threshold.
5. The method of claim 3 , wherein re-localizing the user is further based on a determination that the scoring criteria exceeds the predetermined threshold for a predetermined duration of time.
6. The method of claim 1 , wherein the one or more detected landmarks comprises one or more of points, lines, planes, point cluster, or meshes.
7. The method of claim 1 , wherein the landmarks in the landmark database are indexed with respect to locations or timestamps.
8. A system comprising: one or more processors and one or more computer-readable non-transitory storage media coupled to one or more of the processors, the one or more computer-readable non-transitory storage media comprising instructions operable when executed by one or more of the processors to cause the system to perform operations comprising:
during a first tracking session:
detect first landmarks in a first image of an environment surrounding a user; and
determine a first location of the user by comparing the first landmarks to landmarks in a landmark database;
during a second tracking session:
capture motion data using a motion sensor; and
estimate, without using the landmark database, a second location of the user based on the motion data and the first location;
detect second landmarks in a second image captured from a viewpoint of the user at the second location;
access, from the landmark database, expected landmarks visible from the viewpoint of the user at the second location;
determine, based on a scoring criteria, that the estimated second location of the user is inaccurate by comparing the expected landmarks with the second landmarks; and
re-localize the user by comparing the landmarks in the landmark database and third landmarks detected in a third image.
9. The system of claim 8 , wherein the detection of second landmarks in a second image captured from a viewpoint of the user at the second location comprises:
for each of the one or more expected landmarks visible from the viewpoint of the user at the second location, searching for a corresponding landmark within the one or more detected second landmarks in the second image.
10. The system of claim 8 , wherein the scoring criteria are based on semantic information associated with the expected landmarks.
11. The system of claim 8 , wherein the re-localization of the user is further based on a determination that the scoring criteria exceeds a predetermined threshold.
12. The system of claim 11 , wherein the re-localization of the user is further based on a determination that the scoring criteria exceeds the predetermined threshold for a predetermined duration of time.
13. The system of claim 8 , wherein the one or more detected landmarks comprises one or more of points, lines, planes, point cluster, or meshes.
14. The system of claim 8 , wherein the landmarks in the landmark database are indexed with respect to locations or timestamps.
15. One or more computer-readable non-transitory storage media embodying software that is operable when executed to cause one or more processors to perform operations comprising:
during a first tracking session:
detect first landmarks in a first image of an environment surrounding a user; and
determine a first location of the user by comparing the first landmarks to landmarks in a landmark database;
during a second tracking session:
capture motion data using a motion sensor; and
estimate, without using the landmark database, a second location of the user based on the motion data and the first location;
detect second landmarks in a second image captured from a viewpoint of the user at the second location;
access, from the landmark database, expected landmarks visible from the viewpoint of the user at the second location;
determine, based on a scoring criteria, that the estimated second location of the user is inaccurate by comparing the expected landmarks with the second landmarks; and
re-localize the user by comparing the landmarks in the landmark database and third landmarks detected in a third image.
16. The media of claim 15 , wherein the detection of second landmarks in a second image captured from a viewpoint of the user at the second location comprises:
for each of the one or more expected landmarks visible from the viewpoint of the user at the second location, searching for a corresponding landmark within the one or more detected second landmarks in the second image.
17. The media of claim 15 , wherein the scoring criteria are based on semantic information associated with the expected landmarks.
18. The media of claim 15 , wherein the re-localization of the user is further based on a determination that the scoring criteria exceeds a predetermined threshold.
19. The media of claim 18 , wherein the re-localization of the user is further based on a determination that the scoring criteria exceeds the predetermined threshold for a predetermined duration of time.
20. The media of claim 15 , wherein the one or more detected landmarks comprises one or more of points, lines, planes, point cluster, or meshes.