Fault-tolerance to provide robust tracking for autonomous and non-autonomous positional awareness
The described positional awareness techniques employing visual-inertial sensory data gathering and analysis hardware with reference to specific example implementations implement improvements in the use of sensors, techniques and hardware design that can enable specific embodiments to provide positional awareness to machines with improved speed and accuracy.
1 . A method of guiding a mobile platform in an unmapped area, including:
selecting two or more tracking states from a set of time dependent tracking states;
detecting when one or more visual sensors has failed or is not updating or is insufficiently updating and whenever one or more visual sensors has failed or is not updating or is insufficiently updating, using a motion detector to control pose estimation by tracking a set of inertial readings from the motion detector, using variance obtained for a plurality of dimensions in the set of inertial readings;
updating current poses for at least two tracking states of the two or more tracking states within the set of time dependent tracking states based upon the set of inertial readings from the motion detector as tracked; and
guiding the mobile platform using at least one of the updated current poses determined for the at least two tracking states.
2 . The method of claim 1 , wherein time dependent tracking states of the set of time dependent tracking states include: (i) a pose and (ii) one or more frames of sensor readings, wherein time dependent tracking states include at least some frames from visual sensors.
3 . The method of claim 2 , wherein at least some frames include sets of 2D feature points located using image information from the visual sensors, and the method further including:
creating a 2D feature correspondences set comprising of common 2D feature points among the at least some frames from visual sensors from the tracking states selected;
triangulating 2D feature points from the 2D feature correspondences set to form a 3D point set;
selecting a subset of the 3D point set that includes 3D points having re-projection error within a threshold; and
updating current poses for at least two of the tracking states within the set of time dependent tracking states subject to a criterion including reduction of a total of re-projection errors of the 3D points as selected.
4 . The method of claim 3 , wherein triangulating further includes selecting 3D points having a triangulation error less than a determined number of pixels.
5 . The method of claim 3 , further including adjusting a threshold for selecting 3d points to control propagation based upon a quantity of 3D points available to update poses in the set of time dependent tracking states.
6 . The method of claim 3 , further configured for performing dynamically determining a window size including a number of tracking states in the set of time dependent tracking states by maintaining a fixed manageable window size depending upon a parameter of at least one sensor.
7 . The method of claim 3 , further configured for performing dynamically determining a window size including a number of tracking states in the set of time dependent tracking states by maintaining a range of time spanning using the window size determined.
8 . The method of claim 3 , further configured for performing dynamically determining a window size including a number of tracking states in the set of time dependent tracking states by maintaining a manageable number of 3D points.
9 . The method of claim 8 , further configured for performing maintaining a number of 3D points in tracking states using a window size of less than 200.
10 . The method of claim 3 , further configured for performing refining 3D points using a least squares function.
11 . The method of claim 3 , further configured for performing: detecting a number of features in an image; and dynamically adjusting an exposure value of at least one visual sensor whenever a lack of features the image is detected.
12 . The method of claim 3 , further configured for performing: calculating a location of at least one 2D feature in an image using optical flow; reducing features to features existing along an epipolar line based upon a location calculated and a prediction of a camera pose; checking for similarity between descriptors of the feature and descriptors for a corresponding feature from a previous frame; determining whether number of features through optical flow is less than a number of features detected in a current frame, and whenever the number of features through optical flow is less than a number of features, detecting 2D corner features directly from an image at image regions without features; bin the image spatially, and adjust a number of features in each bin so that the number of features in the bins is balanced.
13 . The method of claim 12 , further configured for performing: linking features in different images captured at different times by a same camera.
14 . The method of claim 12 , further configured for performing: correlating unmatched features for a camera, by:
collecting a set of features that are not matched to any other features by existing links; collecting a set of available links that are do not match to a feature from the camera in a current time frame; calculating a similarity score between each member in the set of features and each member in the set of available links; wherein similarity between a feature and a link is a best similarity score between a descriptor of the feature and descriptors of features in the link; and when the similarity score is greater than a threshold, adding the feature to the link.
15 . The method of claim 1 , further including tracking inertial readings using variance obtained for a plurality of dimensions in the set of inertial readings by:
obtaining a computed variance for each dimension of a total of 3 dimensions of the set of inertial readings;
determining whether the mobile platform is moving or at rest based upon the variance computed; and
whenever the mobile platform is determined to be moving, performing extended Kalman filtering on a set of time dependent tracking states to determine pose information from the set of time dependent tracking states and whenever the mobile platform is determined to be static providing a computed pose whenever the mobile platform is determined to be at rest.
16 . The method of claim 15 , wherein the computed pose is determined by selectively performing at least one of:
(i) propagating a zero velocity applied to location information of a pose and change orientation information determined for the mobile platform; and
(ii) adding a noise factor to a previous pose.
17 . The method of claim 15 , wherein a variance above a threshold indicates a moving platform; and a variance below a threshold indicates a static platform.
18 . The method of claim 1 , further configured for performing: detecting a number of inertial measurement updates occurring after a successful image update; and switching to gyroscopic sensor propagation after N consecutive inertial measurement updates are detected.
19 . The method of claim 1 , further configured for performing synchronizing time associated with at least one of the set of time dependent tracking states with a time maintained by a controller controlling the mobile platform, by updating the time associated with the time associated with at least one of the a set of time dependent tracking states by (i) a constant offset and (ii) a transfer delay.
20 . A non-transitory computer readable medium having instructions stored thereon for performing a method of guiding a mobile platform in an unmapped area and without producing a map of the unmapped area, which instructions when executed by a processor perform actions including:
selecting two or more tracking states from a set of time dependent tracking states;
detecting when one or more visual sensors has failed or is not updating or is insufficiently updating and whenever one or more visual sensors has failed or is not updating or is insufficiently updating, using a motion detector to control pose estimation by tracking a set of inertial readings from the motion detector, using variance obtained for a plurality of dimensions in the set of inertial readings;
updating current poses for at least two tracking states of the two or more tracking states within the set of time dependent tracking states based upon the set of inertial readings from the motion detector as tracked; and
guiding the mobile platform using at least one of the updated current poses determined for the at least two tracking states.