Multi-Sensor Position and Orientation Determination System and Device

1 . A device comprising:

an inertial measurement unit (IMU) sensor configured to generate IMU data of the device;

a camera configured to generate a plurality of video frames;

a radio-based sensor configured to generate radio-based sensor data based on an absolute reference frame relative to the device; and

at least one hardware processor comprising a visual inertial navigation (VIN) application, the VIN application being configured to perform operations comprising:

synchronize the plurality of video frames with the IMU data using a reference clock source of the radio-based sensor, the reference clock source controlling both the time data from the plurality of video frames and the IMU data;

compute a first estimated spatial state of the device based on the synchronized plurality of video frames with the IMU data;

compute a second estimated spatial state of the device based on the radio-based sensor data; and

determine a spatial state of the device based on a combination of the first and second estimated spatial states of the device.

2 . The device of claim 1 , wherein the operations further comprise:

detect and track at least one feature in a video sequence of the plurality of video frames;

match the at least one feature between adjacent video frames to detect inliers;

detect outliers over a sliding window of video frames of the plurality of video frames using the IMU data; and

compute the first estimated spatial state of the device based on detecting the outliers and the inliers, wherein the spatial state of the device includes a position, an orientation, and a velocity of the device.

3 . The device of claim 1 , wherein the operations further comprise:

compute the first estimated spatial state of the device based on the synchronized plurality of video frames with the IMU data for a period of time during which the device is without access to the radio-based sensor data;

access a second radio-based sensor data generated after the period of time;

compute the second estimated spatial state of the device based on the second radio-based sensor data;

adjust the first estimated spatial state of the device based on the second estimated spatial state of the device;

access the reference clock source of the sensor-based sensor after the period of time; and

adjust a clock of the IMU sensor based on the reference clock source.

4 . The device of claim 3 , wherein the IMU sensor operates at a refresh rate higher than that of the camera, and wherein the radio-based sensor comprises at least one of a GPS sensor and a wireless sensor.

5 . The device of claim 2 , wherein the operations further comprise:

determine a historical trajectory of the device based on the combination of the first and second estimated spatial states of the device.

6 . The device of claim 1 , wherein the operations further comprise:

generate and position augmented reality content in a display of the device based on the spatial state of the device.

7 . The device of claim 6 , wherein the operations further comprise:

calibrate the camera offline for focal length, principal point, pixel aspect ratio, and lens distortion, to calibrate the IMU sensor for noise, scale, and bias.

8 . The device of claim 1 , wherein the IMU data comprises an angular rate of change and a linear acceleration.

9 . The device of claim 2 , wherein the feature comprises predefined stationary interest points and line features.

10 . The device of claim 2 , wherein the operations further comprise:

update the spatial state of the device based on every video frame from the camera in real time; and

adjust a position of augmented reality content in a display of the device based on a latest spatial state of the device.

11 . A computer-implemented method comprising:

accessing inertial measurement unit (IMU) data from at least one IMU sensor of a device;

accessing a plurality of video frames from a camera of the device;

accessing radio-based sensor data from a radio-based sensor, the radio-based sensor data based on an absolute reference frame relative to the device;

synchronizing the plurality of video frames with the IMU data using a reference clock source of the radio-based sensor, the reference clock source controlling both the time data from the plurality of video frames and the IMU data;

computing a first estimated spatial state of the device based on the synchronized plurality of video frames with the IMU data;

computing a second estimated spatial state of the device based on the radio-based sensor data; and

determining a spatial state of the device based on a combination of the first and second estimated spatial states of the device.

12 . The computer-implemented method of claim 11 , further comprising:

detecting and tracking at least one feature in a video sequence of the plurality of video frames;

matching the at least one feature between adjacent video frames to detect inliers;

detecting outliers over a sliding window of video frames of the plurality of video frames using the IMU data; and

computing the first estimated spatial state of the device based on detecting the outliers and the inliers, wherein the spatial state of the device includes a position, an orientation, and a velocity of the device.

13 . The computer-implemented method of claim 11 , further comprising:

computing the first estimated spatial state of the device based on the synchronized plurality of video frames with the IMU data for a period of time during which the device is without access to the radio-based sensor data;

accessing a second radio-based sensor data generated after the period of time;

computing the second estimated spatial state of the device based on the second radio-based sensor data;

adjusting the first estimated spatial state of the device based on the second estimated spatial state of the device;

accessing the reference clock source of the radio-based sensor after the period of time; and

adjusting a clock of the IMU sensor based on the reference clock source.

14 . The computer-implemented method of claim 11 , wherein the IMU sensor operates at a refresh rate higher than that of the camera.

15 . The computer-implemented method of claim 13 , further comprising:

determining a historical trajectory of the device based on the combination of the first and second estimated spatial states of the device.

16 . The computer-implemented method of claim 11 , further comprising:

generating and positioning augmented reality content in a display of the device based on the spatial state of the device.

17 . The computer-implemented method of claim 16 , further comprising:

calibrating the camera offline for focal length, principal point, pixel aspect ratio, and lens distortion;

calibrating the IMU sensor for noise, scale, and bias.

18 . The computer-implemented method of claim 11 , wherein the IMU data comprises an angular rate of change and a linear acceleration.

19 . The computer-implemented method of claim 12 , wherein the feature comprises predefined stationary interest points and line features.

20 . A non-transitory machine-readable storage medium, tangibly embodying a set of instructions that, when executed by at least one processor, causes the at least one processor to perform a set of operations comprising:

accessing inertial measurement unit (IMU) data from at least one IMU sensor of a device;

accessing a plurality of video frames from a camera of the device;

accessing radio-based sensor data from a radio-based sensor, the radio-based sensor data based on an absolute reference frame relative to the device;

synchronizing the plurality of video frames with the IMU data using a reference clock source of the radio-based sensor, the reference clock source controlling both the time data from the plurality of video frames and the IMU data;

computing a first estimated spatial state of the device based on the synchronized plurality of video frames with the IMU data;

computing a second estimated spatial state of the device based on the radio-based sensor data; and

determining a spatial state of the device based on a combination of the first and second estimated spatial states of the device.