Visual-inertial positional awareness for autonomous and non-autonomous device

1. A system including:

a mobile platform controllable by a host, the mobile platform having disposed thereon:

a visual sensor comprising at least one RGB sensing capable camera and at least one other grayscale camera disposed at a distance relative to one another to form a region in which the fields of view at least partially overlap, thereby providing stereoscopic imaging capability;

a multi-axis inertial measuring unit (IMU) including one or more accelerometer or gyroscopic sensors capable of providing measurement of at least acceleration; and

a visual inertial control unit, including:

a first interface that couples to the visual sensor to receive sets of image data;

a second interface that couples to the multi-axis IMU to receive accelerometer sensor data;

a storage that stores the sets of image data;

one or more processors having access to the storage;

an inertial measurement engine including instructions that when executed by the one or more processors perform time stamping of inertial data received via the second interface, corrects the timestamped inertial data for bias, applies a stored scale factor to the corrected inertial data and corrects the scaled inertial data for misalignment in the IMU to form localization data;

an imaging engine including instructions that when executed by the one or more processors perform imaging undistortion on the sets of image data; and

a communications interface to provide the localization data and the undistorted sets of image data to a host controlling the mobile platform.

2. The system of claim 1 , wherein the visual inertial control unit further:

detects features in the undistorted sets of images; and

provides the detected features to the host via the communications interface.

3. The system of claim 2 , wherein features are detected using Shi-Tomasi feature detection.

4. The system of claim 2 , wherein the imaging engine further:

determines feature descriptors for the detected features; and

provides the feature descriptors to the host via the communications interface.

5. The system of claim 2 , wherein the imaging engine further:

determines feature correspondences between features detected from successive images in the sets of images; and

provides the feature correspondences to the host via the communications interface.

6. The system of claim 5 , wherein feature correspondences are determined using optical flow.

7. The system of claim 1 , wherein the imaging engine further:

generates a bag of words description for an image frame to be used in re-localization.

8. An apparatus for guiding a mobile device using information from one or more cameras with distance calculation and multi-axis inertial measuring unit (IMU), the apparatus including:

a first interface that couples to the one or more cameras to receive sets of image data;

a second interface that couples to a multi-axis IMU to receive accelerometer sensor data;

a storage that stores the sets of image data;

one or more processors having access to the storage;

an inertial measurement engine including instructions that when executed by the one or more processors perform time stamping of inertial data received via the second interface, corrects inertial readouts in the timestamped inertial data for bias, applies a stored scale factor to the corrected inertial data and corrects the scaled inertial data for misalignment in the IMU to form localization data;

an imaging engine including instructions that when executed by one or more processors perform imaging undistortion on the sets of image data; and

a communications interface to provide the localization data and the undistorted sets of image data to a host controlling the mobile device.

9. The apparatus of claim 8 , wherein the imaging engine further:

detects features in the undistorted sets of images; and

provides the detected features to the host via the communications interface.

10. The apparatus of claim 9 , wherein features are detected using Shi-Tomasi feature detection.

11. The apparatus of claim 9 , wherein the imaging engine further:

determines feature descriptors for the detected features; and

provides the feature descriptors to the host via the communications interface.

12. The apparatus of claim 9 , wherein the imaging engine further:

determines feature correspondences between features detected from successive images in the sets of images; and

provides the feature correspondences to the host via the communications interface.

13. The apparatus of claim 12 , wherein feature correspondences are determined using optical flow.

14. The apparatus of claim 8 , wherein the imaging engine further:

generates a bag of words description for an image frame to be used in re-localization.

15. A non-transitory computer readable storage medium impressed with computer program instructions to guide a mobile device using information from a camera with distance calculation and multi-axis inertial measuring unit (IMU), the instructions, when executed on a processor, implement a method, the method including:

buffering image sets from a visual sensor comprising at least one visible light sensing capable camera and from at least one other camera disposed at a distance relative to one another to form a region in which the fields of view at least partially overlap, thereby providing stereoscopic imaging capability;

buffering inertial measurements from a multi-axis inertial measuring unit (IMU) capable of providing measurement of at least acceleration;

receiving at a visual inertial control unit the sets of image data;

receiving at the visual inertial control unit sensor data from the multi-axis IMU;

time stamping by an inertial measurement engine the inertial data received;

correcting inertial readouts in the timestamped inertial data for bias;

scaling the inertial readouts using a stored scale factor the corrected inertial data;

correcting the scaled inertial data for misalignment in the IMU to form localization data;

performing imaging undistortion on the sets of image data; and

providing across a communications interface the localization data and the undistorted sets of image data to a host controlling a mobile platform.

16. The non-transitory computer readable storage medium of claim 15 , implementing the method further including:

detecting features in the undistorted sets of images; and

providing the detected features to the host via the communications interface.

17. The non-transitory computer readable storage medium of claim 16 , wherein the features are detected using Shi-Tomasi feature detection.

18. The non-transitory computer readable storage medium of claim 16 , implementing the method further including:

determining feature descriptors for the detected features; and

providing the feature descriptors to the host via the communications interface.

19. The non-transitory computer readable storage medium of claim 16 , implementing the method further including:

determining feature correspondences between features detected from successive images in the sets of images; and

providing the feature correspondences to the host via the communications interface.

20. The non-transitory computer readable storage medium of claim 19 , wherein feature correspondences are determined using optical flow.

21. The non-transitory computer readable storage medium of claim 15 , implementing the method further including:

generating a bag of words description for an image frame to be used in re-localization.