Visual-inertial positional awareness for autonomous and non-autonomous device

1. A system including:

a mobile platform controllable by a host, the mobile platform having disposed thereon:

a visual sensor comprising at least one RGB sensing capable camera and at least one other grayscale camera disposed at a distance relative to one another to form a region in which the fields of view at least partially overlap, thereby providing stereoscopic imaging capability;

a multi-axis inertial measuring unit (IMU) capable of providing measurement of at least acceleration; and

a visual inertial control unit, including:

a first interface that couples to the visual sensor to receive sets of image data;

a second interface that couples to the multi-axis IMU to receive accelerometer sensor data;

a cache storage that stores the sets of image data;

a single instruction-multiple data processing element having direct memory access to the cache storage;

an inertial measurement engine that performs time stamping of inertial data received via the second interface, corrects the timestamped inertial data for bias, applies a stored scale factor to the corrected inertial data and corrects the scaled inertial data for misalignment in the IMU to form localization data;

an imaging engine that performs imaging undistortion on the sets of image data; and

a communications interface to provide the localization data and the undistorted sets of image data to a host controlling the mobile platform.

2. The system of claim 1 , wherein the visual inertial control unit further:

detects features in the undistorted sets of images; and

provides the detected features to the host via the communications interface.

3. The system of claim 2 , wherein features are detected using Shi-Tomasi feature detection.

4. The system of claim 2 , wherein the imaging engine further:

determines feature descriptors for the detected features; and

provides the feature descriptors to the host via the communications interface.

5. The system of claim 2 , wherein the imaging engine further:

determines feature correspondences between features detected from successive images in the sets of images; and

provides the feature correspondences to the host via the communications interface.

6. The system of claim 5 , wherein feature correspondences are determined using optical flow.

7. The system of claim 1 , wherein the imaging engine further:

generates a bag of words description for an image frame to be used in re-localization.

8. An apparatus for guiding a mobile device using information from one or more cameras with distance calculation and multi-axis inertial measuring unit (IMU), the apparatus including:

a first interface that couples to the one or more cameras to receive sets of image data;

a second interface that couples to a multi-axis IMU to receive accelerometer sensor data;

a cache storage that stores the sets of image data;

a single instruction-multiple data processing element having direct memory access to the cache storage;

an inertial measurement engine that performs time stamping of inertial data received via the second interface, corrects inertial readouts in the timestamped inertial data for bias, applies a stored scale factor to the corrected inertial data and corrects the scaled inertial data for misalignment in the IMU to form localization data;

an imaging engine that performs imaging undistortion on the sets of image data; and

a communications interface to provide the localization data and the undistorted sets of image data to a host controlling the mobile device.

9. The apparatus of claim 8 , wherein the apparatus further includes the imaging engine:

detects features in the undistorted sets of images; and

provides the detected features to the host via the communications interface.

10. The apparatus of claim 9 , wherein features are detected using Shi-Tomasi feature detection.

11. The apparatus of claim 9 , wherein the imaging engine further:

determines feature descriptors for the detected features; and

provides the feature descriptors to the host via the communications interface.

12. The apparatus of claim 9 , wherein the imaging engine further:

determines feature correspondences between features detected from successive images in the sets of images; and

provides the feature correspondences to the host via the communications interface.

13. The apparatus of claim 12 , wherein feature correspondences are determined using optical flow.

14. The apparatus of claim 8 , wherein the imaging engine further:

generates a bag of words description for an image frame to be used in re-localization.

15. A non-transitory computer readable storage medium impressed with computer program instructions to guide a mobile device using information from a camera with distance calculation and multi-axis inertial measuring unit (IMU), the instructions, when executed on a processor, implement a method, the method including:

buffering image sets from a visual sensor comprising at least one RGB sensing capable camera and from at least one other grayscale camera disposed at a distance relative to one another to form a region in which the fields of view at least partially overlap, thereby providing stereoscopic imaging capability;

buffering inertial measurements from a multi-axis inertial measuring unit (IMU) capable of providing measurement of at least acceleration;

receiving at a visual inertial control unit the sets of image data;

receiving at the visual inertial control unit sensor data from the multi-axis IMU;

time stamping by an inertial measurement engine the inertial data received,

correcting inertial readouts in the timestamped inertial data for bias,

scaling the inertial readouts using a stored scale factor the corrected inertial data;

correcting the scaled inertial data for misalignment in the IMU to form localization data;

performing imaging undistortion on the sets of image data; and

providing across a communications interface the localization data and the undistorted sets of image data to a host controlling a mobile platform.

16. The non-transitory computer readable storage medium of claim 15 , implementing the method further including:

detecting features in the undistorted sets of images; and

providing the detected features to the host via the communications interface.

17. The non-transitory computer readable storage medium of claim 16 , wherein the features are detected using Shi-Tomasi feature detection.

18. The non-transitory computer readable storage medium of claim 16 , implementing the method further including:

determining feature descriptors for the detected features; and

providing the feature descriptors to the host via the communications interface.

19. The non-transitory computer readable storage medium of claim 16 , implementing the method further including:

determining feature correspondences between features detected from successive images in the sets of images; and

providing the feature correspondences to the host via the communications interface.

20. The non-transitory computer readable storage medium of claim 19 , wherein feature correspondences are determined using optical flow.

21. The non-transitory computer readable storage medium of claim 15 , implementing the method further including:

generating a bag of words description for an image frame to be used in re-localization.