Visual-inertial positional awareness for autonomous and non-autonomous device

1. A system including:

a map server including computer instructions that when executed by a processor coupled to a memory store multi-layer maps of an area under exploration using information sourced by one or more autonomous units; and

one or more autonomous units selected from a set comprising autonomous vehicles and robots, including a first autonomous unit, each autonomous unit including at least a quad camera visual sensor and at least one selected from a global positioning system and an inertial measurement unit; and each autonomous unit configured to perform:

receiving a proto-roadmap including only roads;

in the autonomous unit, capturing a set of keyrigs, each keyrig is a set of quad images with a pose generated using combinations of global positioning system, inertial measurement unit, and visual information of a scene by the first autonomous unit during travel along one of the roads in the proto-roadmap;

determining a ground perspective view, including at least road marking information for at least one of the roads in the proto-roadmap from the visual information captured;

determining a spatial perspective view, including objects along at least one of the roads in the proto-roadmap from the visual information captured;

classifying objects from the spatial perspective view into moving objects and non-moving objects;

building at least one multi-layer map including a stationary portion consisting of the proto-roadmap, the non-moving objects from the spatial perspective view and the road marking information from the ground perspective view, wherein at least one multi-layer map is accurate within centimeters; and

providing the multi-layer map via a communications link to the map server that stores and distributes multi-layer maps to guide the autonomous unit at a future time and at least one other autonomous unit.

2. The system of claim 1 , further including at least one multi-layer map with an accuracy within 10 centimeters.

3. The system of claim 1 , further including at least one multi-layer map with an accuracy within 5 centimeters.

4. The system of claim 1 , further including providing moving objects to a further process that avoids potential collisions with moving objects and the autonomous unit.

5. The system of claim 1 , further including substantially contemporaneously tracking a position of the autonomous unit against the multi-layer map.

6. The system of claim 1 , further including providing the proto-roadmap, the ground perspective view and the spatial perspective view as layers in a multi-layer map data structure.

7. The system of claim 1 , further including generating several hundred thousand images during one hour of operation by an autonomous unit.

8. The system of claim 1 , further including storing a time of day with the multi-layer map.

9. The system of claim 1 , further including storing a weather condition with the multi-layer map.

10. The system of claim 1 , further including receiving semantic information from the map server and classifying the objects using the semantic information into moving objects and non-moving objects.

11. The system of claim 1 , further including: building a semantic view from the spatial perspective view and objects classified as non-moving objects.

12. The system of claim 1 , further including:

detecting when visual information is insufficient to determine the ground perspective view; and

fall back to providing the proto-roadmap whenever visual information is insufficient.

13. The system of claim 1 , further including:

detecting when visual information is insufficient to determine a spatial perspective view; and

fall back to providing the ground perspective view whenever visual information is insufficient.

14. The system of claim 1 , further including:

detecting when visual or semantic information is insufficient to classify objects; and

fall back to providing the spatial perspective view whenever visual or semantic information is insufficient.

15. The system of claim 1 , further including identifying an object to be included in the spatial perspective view:

extracting a first set of 2D features of the road marking from a first 360-degrees image in a keyrig selected from a set of keyrigs provided by an autonomous unit;

extracting a second set of 2D features of the road marking from a 360-degrees second image in the keyrig selected;

receiving a position of the autonomous unit when the 360-degrees images were captured including longitude and latitude as input;

triangulating the first set of 2D features from the first 360-degrees image and the second set of 2D features from the second 360-degrees image to derive location for feature points of the road marking relative to the position of an autonomous unit; and

generating for at least one feature point of the object, a global position, including longitude, latitude, and height and adding the global position and feature descriptors of the object to the spatial perspective view.

16. The system of claim 1 , further including identifying a road marking to be included in the ground perspective view:

extracting a first set of 2D features of the road marking from a first 360-degrees image in a keyrig selected from a set of keyrigs provided by an autonomous unit;

extracting a second set of 2D features of the road marking from a 360-degrees second image in the keyrig selected;

receiving a position of the autonomous unit when the 360-degrees images were captured including longitude and latitude as input;

triangulating the first set of 2D features from the first 360-degrees image and the second set of 2D features from the second 360-degrees image to derive location for feature points of the road marking relative to the position of an autonomous unit; and

generating for at least one feature point of the road marking, a global position, including longitude and latitude, and adding the global position and feature descriptors of the road marking to the ground perspective view.

17. A method for automatically building multi-layer maps by a first autonomous unit selected from a set comprising autonomous vehicles and robots that includes at least a quad camera visual sensor and at least one selected from a global positioning system and an inertial measurement unit, the method including:

receiving a proto-roadmap including only roads;

in the autonomous unit, capturing a set of keyrigs, each keyrig is a set of quad images with a pose generated using combinations of global positioning system, inertial measurement unit, and visual information of a scene by the first autonomous unit during travel along one of the roads in the proto-roadmap;

determining a ground perspective view, including at least road marking information for at least one of the roads in the proto-roadmap from the visual information captured;

determining a spatial perspective view including objects along at least one of the roads in the proto-roadmap from the visual information captured;

classifying objects from the spatial perspective view into moving objects and non-moving objects;

building at least one multi-layer map including a stationary portion consisting of the proto-roadmap, the non-moving objects from the spatial perspective view and the road marking information from the ground perspective view, wherein at least one multi-layer map is accurate within centimeters; and

providing the multi-layer map via a communications link to a map server including computer instructions that when executed by a processor coupled to a memory that stores and distributes multi-layer maps to guide the autonomous unit at a future time and at least one other autonomous unit.

18. The method of claim 17 , further including:

detecting features in sets of images using Shi-Tomasi feature detection;

determining feature descriptors for the features detected; and

providing at least one of the features detected and the feature descriptors to a host via a communications interface.

19. The method of claim 18 , further including:

determining feature correspondences between features detected from successive images in the sets of images; and

providing the feature correspondences to the host via the communications interface.

20. The method of claim 19 , wherein feature correspondences are determined using optical flow.

21. A non-transitory computer readable storage medium impressed with computer program instructions automatically building multi-layer maps by a first autonomous unit selected from a set comprising autonomous vehicles and robots that includes at least a quad camera visual sensor and at least one selected from a global positioning system and an inertial measurement unit, which instructions, when executed on a processor, perform:

receiving a proto-roadmap including only roads;

in the autonomous unit, capturing a set of keyrigs, each keyrig is a set of quad images with a pose generated using combinations of global positioning system, inertial measurement unit, and visual information of a scene by the first autonomous unit during travel along one of the roads in the proto-roadmap;

determining a ground perspective view, including at least road marking information for at least one of the roads in the proto-roadmap from the visual information captured;

determining a spatial perspective view including objects along at least one of the roads in the proto-roadmap from the visual information captured;

classifying objects from the spatial perspective view into moving objects and non-moving objects;

building at least one multi-layer map including a stationary portion consisting of the proto-roadmap, the non-moving objects from the spatial perspective view and the road marking information from the ground perspective view, wherein at least one multi-layer map is accurate within centimeters; and

providing the multi-layer map via a communications link to a map server including computer instructions that when executed by a processor coupled to a memory that stores and distributes multi-layer maps to guide the autonomous unit at a future time and at least one other autonomous unit.