Distributed aperture visual inertia navigation
A system and method for visual inertial navigation for augmented reality are described. In some embodiments, at least one camera of a wearable device generates a plurality of video frames. At least one inertial measurement unit (IMU) sensors of the wearable device generates IMU data. Features in the plurality of video frames for each camera are tracked. The plurality of video frames for each camera are synchronized and aligned based on the IMU data. A dynamic state of the wearable device is computed based on the synchronized plurality of video frames with the IMU data for each camera. Augmented reality content is generated and positioned in a display of the wearable device based on the dynamic state of the wearable device.
1. A wearable device comprising:
at least one camera configured to capture an image and generate a plurality of video frames;
at least one inertial measurement unit (IMU) sensor configured to generate IMU data of the wearable device;
a feature tracking module configured to track at least one feature in the plurality of video frames for each camera;
a synchronization module configured to synchronize and align the plurality of video frames for each camera based on the IMU data;
a visual inertial navigation (VIN) module, executable by at least one hardware processor, configured to compute a dynamic state of the wearable device based on the synchronized plurality of video frames with the IMU data; and
an augmented reality content module configured to generate and position augmented reality content in a display of the wearable device based on the dynamic state of the wearable device.
2. The wearable device of claim 1 , further comprising:
a calibration module configured to calibrate the at least one camera off-line for focal length, principal point, pixel aspect ratio, lens distortion, and to calibrate the at least one IMU sensor for noise, scale, and bias, and to apply the calibration information to the plurality of video frames and the IMU data.
3. The wearable device of claim 1 , wherein the IMU data comprises an angular rate of change and a linear acceleration.
4. The wearable device of claim 1 , wherein the features comprises predefined stationary interest points and line features.
5. The wearable device of claim 1 , wherein the dynamic state of the wearable device comprises position data, orientation data, three-dimensional geometry data, gyroscope data, and accelerometer bias and scale data.
6. The wearable device of claim 1 , wherein the VIN module updates the dynamic state on every frame from the at least camera in real-time.
7. The wearable device of claim 1 , further comprising:
a global refinement module configured to access historical dynamic states from the wearable device to refine real-time state estimates from the IMU sensor.
8. The wearable device of claim 1 , further comprising:
a propagation module configured to adjust a position of the augmented reality content in the display based on a latest dynamic state of the wearable device.
9. The wearable device of claim 1 , wherein the augmented reality content comprises three-dimensional graphical content.
10. The wearable device of claim 1 , wherein the display of the wearable device includes a transparent visor of a helmet.
11. A computer-implemented method comprising:
generating a plurality of video frames with at least one camera of a wearable device;
generating inertial measurement unit (IMU) data with at least one IMU sensor of the wearable device;
tracking at least one feature in the plurality of video frames for each camera;
synchronizing and aligning the plurality of video frames for each camera based on the IMU data;
computing, by a hardware processor of a machine, a dynamic state of the wearable device based on the synchronized plurality of video frames with the IMU data; and
generating and positioning augmented reality content in a display of the wearable device based on the dynamic state of the wearable device.
12. The computer-implemented method of claim 11 , further comprising:
calibrating the at least one camera off-line for focal length, principal point, pixel aspect ratio, lens distortion;
calibrating the at least one IMU sensor for noise, scale, and bias; and
applying the calibration information to the plurality of video frames and the IMU data.
13. The computer-implemented method of claim 11 , wherein the IMU data comprises an angular rate of change and a linear acceleration.
14. The computer-implemented method of claim 11 , wherein the features comprises predefined stationary interest points and line features.
15. The computer-implemented method of claim 11 , wherein the dynamic state of the wearable device comprises position data, orientation data, three-dimensional geometry data, gyroscope data, and accelerometer bias and scale data.
16. The computer-implemented method of claim 11 , further comprising:
updating the dynamic state on every frame from the at least camera in real-time.
17. The computer-implemented method of claim 11 , further comprising:
accessing historical dynamic states from the wearable device to refine real-time state estimates from the IMU sensor.
18. The computer-implemented method of claim 11 , further comprising:
adjusting a position of the augmented reality content in the display based on a latest dynamic state of the wearable device.
19. The computer-implemented method of claim 11 , wherein the augmented reality content comprises three-dimensional graphical content, the display of the wearable device including a transparent visor of a helmet, a location and size of a projected image of the three-dimensional graphical content adjusted in the transparent visor based on the dynamic state of the wearable device.
20. A non-transitory machine-readable storage medium, tangibly embodying a set of instructions that, when executed by at least one processor, causes the at least one processor to perform a set of operations comprising:
generating a plurality of video frames with at least one camera of a wearable device;
generating inertial measurement unit (IMU) data with at least one IMU sensor of the wearable device;
tracking at least one feature in the plurality of video frames for each camera;
synchronizing and aligning the plurality of video frames for each camera based on the IMU data;
computing a dynamic state of the wearable device based on the synchronized plurality of video frames with the IMU data; and
generating and positioning augmented reality content in a display of the wearable device based on the dynamic state of the wearable device.