IP Library Granted Patent US 11,740,321
Granted Patent B2
US 11,740,321 · App. 15/926,557 · Granted Aug 29, 2023

Visual inertial odometry health fitting

Inventors: Oleg Naroditsky (San Francisco, CA); Kuen-Han Lin (Mountain View, CA); Dimitrios Kottas (Sunnyvale, CA)
Assignee: Apple Inc.
G01S5/16G01S19/47G06F18/217G06F18/24133G06N3/042G06T7/251G06V10/764G01S19/52G06F2218/12G06V2201/03
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,740,321
App. No.
15/926,557
Granted
Aug 29, 2023
Kind
B2
Abstract

Systems, methods, and computer readable media to track and estimate the accuracy of a visual inertial odometry (VIO) system. Various embodiments are able to receive one or more VIO feature measurements associated with a set of image frames from a VIO system and generate a plurality of feature models to estimate health values for the VIO system. The various embodiments determine a plurality of feature health values with the feature models based on the VIO feature measurements and compare the feature health values with ground truth health scores associated with the set of image frames to determine one or more errors. The feature model parameters are updated based on the comparison with the feature health values with ground truth health scores.

Claims (50)

1. A non-transitory program storage device, readable by one or more programmable control devices and comprising instructions stored thereon to cause the one or more programmable control devices to:

receive one or more visual inertial odometry (VIO) feature measurements associated with each image frame of a set of test image frames from a VIO system, wherein the VIO feature measurements each correspond to a measurement of features detected by the VIO system in processing the set of test image frames to determine location information for the VIO system;

generate a plurality of feature models of a neural network classifier model to estimate health values for the VIO system for a particular feature measurement of the one or more VIO feature measurements, wherein each feature model includes at least one feature model parameter;

determine a plurality of feature health values with the feature models based on the one or more VIO feature measurements, wherein each of the plurality of feature health values correspond to an accuracy of a corresponding feature measurement associated with the feature model;

compare the feature health values with ground truth health scores associated with the set of test image frames to determine one or more errors between the feature health values and the ground truth health scores, wherein the ground truth health scores comprise a set of predetermined health scores corresponding to the VIO system and the set of test image frames; and

update the feature model parameters for one or more of the feature models based on the one or more errors.

2. The non-transitory program storage device of claim 1 , wherein the instructions further cause the one or more programmable control devices to:

determine a plurality of frame health scores based on one or more feature health values and a frame model, wherein the frame model includes at least one frame model parameter;

determine an overall health score for the VIO system based on the frame health scores and a decaying weight model, wherein the decaying weight model includes at least one weight model parameter that weights the frame health scores;

compare the frame health scores and the overall health score with a second set of ground truth health scores associated with the set of test image frames; and

update the frame model parameter and the weight model parameter based on the comparison of the frame health scores and the overall health score with the second set of ground truth health scores.

3. The non-transitory program storage device of claim 2 , wherein each frame health score corresponds to one image frame within the set of test image frames.

4. The non-transitory program storage device of claim 2 , wherein the at least one weight model parameter is configured to provide a greater weight value for images frames within the set of test image frames that are relatively more current and provides a smaller weight value for image frames within the set of test image frames that are relatively older.

5. The non-transitory program storage device of claim 2 , wherein each frame health score corresponds to at least some image frames within the set of test image frames.

6. The non-transitory program storage device of claim 2 , wherein the frame model, the feature models, and the decaying weight model are part of the neural network classifier model.

7. The non-transitory program storage device of claim 6 , wherein the neural network classifier model is a two-layer classifier model.

8. The non-transitory program storage device of claim 1 , wherein the feature models are sigmoid models.

9. The non-transitory program storage device of claim 1 , wherein the feature models are 1-dimensional filters.

10. A system comprising:

an image capture device;

memory; and

one or more programmable control devices operable to interact with the image capture device and the memory, and to perform operations comprising:

receiving one or more visual inertial odometry (VIO) feature measurements associated with each image frame of a set of test image frames from a VIO system, wherein the VIO feature measurements each correspond to a measurement of features detected by the VIO system in processing the set of test image frames to determine location information for the VIO system;

generating a plurality of feature models of a neural network classifier model to estimate health values for the VIO system for a particular feature measurement of the one or more VIO feature measurements, wherein each feature model includes at least one feature model parameter;

determining a plurality of feature health values with the feature models based on the one or more VIO feature measurements, wherein each of the plurality of feature health values correspond to an accuracy of a corresponding feature measurement associated with the feature model;

comparing the feature health values with ground truth health scores associated with the set of test image frames to determine one or more errors between the feature health values and the ground truth health scores, wherein the ground truth health scores comprise a set of predetermined health scores corresponding to the VIO system and the set of test image frames; and

updating the feature model parameters for one or more of the feature model based on the one or more errors.

11. The system of claim 10 , wherein the one or more programmable control devices further performs operations comprising:

determining a plurality of frame health scores based on one or more feature health values and a frame model, wherein the frame model includes at least one frame model parameter;

determining an overall health score for the VIO system based on the frame health scores and a decaying weight model, wherein the decaying weight model includes at least one weight model parameter that weights the frame health scores;

comparing the frame health scores and the overall health score with a second set of ground truth health scores associated with the set of test image frames; and

updating the frame model parameter and the weight model parameter based on the comparison of the frame health scores and the overall health score with the second set of ground truth health scores.

12. The system of claim 11 , wherein the at least one weight model parameter is configured to provide a greater weight value for images frames within the set of test image frames that are relatively more current and provides a smaller weight value for image frames within the set of test image frames that are relatively older.

13. The system of claim 11 , wherein the frame model, the feature models, and the decaying weight model are part of the neural network classifier model.

14. The system of claim 13 , wherein the neural network classifier model is a two-layer classifier model.

15. The system of claim 11 , wherein each frame health score corresponds to one image frame within the set of test image frames.

16. The system of claim 10 , wherein the feature models are sigmoid models.

17. A method comprising:

receiving one or more visual inertial odometry (VIO) feature measurements associated with each image frame of a set of test image frames from a VIO system, wherein the VIO feature measurements each correspond to a measurement of features detected by the VIO system in processing the set of test image frames to determine location information for the VIO system;

generating, by an electronic device, a plurality of feature models of a neural network classifier model to estimate health values for the VIO system for a particular feature measurement of the one or more VIO feature measurements, wherein each feature model includes at least one feature model parameter;

determining a plurality of feature health values with the feature models based on the one or more VIO feature measurements, wherein each of the plurality of feature health values correspond to an accuracy of a corresponding feature measurement associated with the feature model;

comparing the feature health values with ground truth health scores associated with the set of test image frames to determine one or more errors between the feature health values and the ground truth health scores, wherein the ground truth health scores comprise a set of predetermined health scores corresponding to the VIO system and the set of test image frames; and

updating the feature model parameters for one or more of the feature model based on the one or more errors.

18. The method of claim 17 , further comprising:

determining, using the electronic device, a plurality of frame health scores based on one or more feature health values and a frame model, wherein the frame model includes at least one frame model parameter;

determining, using the electronic device, an overall health score for the VIO system based on the frame health scores and a decaying weight model, wherein the decaying weight model includes at least one weight model parameter that weights the frame health scores;

comparing, using the electronic device, the frame health scores and the overall health score with a second set of ground truth health scores associated with the set of test image frames; and

updating, using the electronic device, the frame model parameter and the weight model parameter based on the comparison of the frame health scores and the overall health score with the second set of ground truth health scores.

19. The method of claim 18 , wherein each frame health score corresponds to at least some image frames within the set of test image frames.

20. The method of claim 17 , wherein the feature models are 1-dimensional filters.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 20, 2018
From: KOTTAS, DIMITRIOS; LIN, KUEN-HAN; NARODITSKY, OLEG
To: APPLE INC.
Reel/Frame 045291/0276 →
Priority Claims (1)
GR 20170100543 · Nov 30, 2017 · national
Continuity (1)
Related Publication 20190164040A1 · May 30, 2019