Systems and methods for generating improved environmental displays for vehicles
View Patent ↗An imaging system for a moving vehicle aggregates pre-existing data with sensor data to provide an image of the surrounding environment in real-time. The pre-existing data are combined with data from one or more 3-D sensors, and 2-D information from a camera, to create a scene model that is rendered for display. The system accepts data from a 3-D sensor, transforms the data into a 3-D data structure, fuses the pre-existing scene data with the 3-D data structure and 2-D image data from a 2-D sensor to create a combined scene model, and renders the combined scene model for display. The system may also weight aspects of data from first and second sensors to select at least one aspect from the first sensor and another aspect from the second sensor; wherein fusing the pre-existing scene data with the sensor data uses the selected aspect from the first sensor and the selected aspect from the second sensor.
1. An imaging system for use in a moving vehicle to provide a synthetic image of an environment in a vicinity of the vehicle in real-time, comprising:
a first data source providing scene data including a priori terrain data and a priori obstacle data;
a second data source providing a 3-D data cloud derived from real-time terrain data and real-time obstacle data;
a third data source providing real-time video image data including terrain image data and obstacle image data;
a scene fusion module configured to receive and combine the a priori terrain data and the a priori obstacle data, the 3-D data cloud, and the real-time video image data to produce a real-time, multi-resolution 3-D scene model including the a priori terrain data and the a priori obstacle data updated in real time by the 3-D data cloud and the video image data;
an image rendering module configured to receive and process the real-time, multi-resolution 3-D scene model to produce an image signal representing the real-time, multi-resolution 3-D scene model; and
a visual display configured to display a virtual view of the environment in the vicinity of the vehicle in response to the image signal.
2. The system of claim 1 , wherein:
the first data source includes a database containing the a priori terrain data and the a priori obstacle data and a first process and classify module configured to yield the scene data as a multi-resolution 3-D a priori terrain and obstacle data structure;
the second data source includes a real-time 3-D environmental sensor and a second process and classify module configured to yield the 3-D data cloud as a real-time 3-D scene data structure;
the third data source includes a camera and a third process and classify module configured to yield the video image data as a real-time video image data structure; and
wherein the scene fusion module is configured to receive and combine the 3-D a priori terrain and obstacle data structure, the real-time 3-D scene data structure, and the real-time video image data structure.
3. The system of claim 2 , wherein the a priori terrain and obstacle data structure includes 2-D and 3-D a priori terrain and obstacle data structures, and the video image data structure includes at least one of a 2-D video image data structure and a 3-D image data structure.
4. The system of claim 2 , wherein at least one of the a priori terrain and obstacle data structure, the real-time 3-D scene data structure, and the video image data structure includes navigation data.
5. The system of claim 2 , wherein the real-time 3-D environmental sensor is selected from the group consisting of one or more of a radar sensor and a lidar sensor.
6. The system of claim 2 , wherein the second data source includes first and second real-time 3-D environmental sensors, each providing real time sensor data as a 3-D data cloud, wherein the second process and classify module is configured to yield the 3-D data cloud from the first and second sensors as first and second real-time 3-D scene data structures, respectively.
7. The system of claim 6 , wherein the scene fusion module is configured to replace at least one aspect of the first real-time 3-D scene data structure sensor with a corresponding aspect of the second real-time 3-D scene data structure.
8. The system of claim 7 , wherein the scene fusion module is configured to compare a quality of the first real-time 3-D scene data structure to a corresponding quality of the second real-time 3-D scene data structure.
9. The system of claim 1 , wherein the scene data provided by the first data source is configured to be updatable by the combined scene model.
10. The system of claim 1 , wherein the scene fusion module is configured to (a) receive a first scene data point set from one of the first, second, and third data sources, (b) receive a second scene data point set from another of the first, second, and third data sources, (c) validate the data points of the second scene data point set against the data points of the first scene data point set to create an updated scene data point set, and (d) produce the real-time, multi-resolution scene model from the updated scene data point set.
11. The system of claim 10 , wherein the first scene data point set is a set of data points, each of which has been determined to have met a predetermined validity criterion, and wherein the scene fusion module is configured to validate the data points of the second scene data point set by determining whether each of the data points of the second scene data point set meets the predetermined validity criterion.
12. The system of claim 11 , wherein the scene fusion module is further configured to use only those data points of the second scene data point set that have been determined to have met the predetermined validity criterion to create the real-time scene data point set.