Thermal imaging for navigation systems and methods
Thermal imaging and navigation systems and related techniques are provided to improve the operation of manned or unmanned mobile platforms, including passenger vehicles. A thermal imaging navigation system includes a thermal imaging system and a logic device configured to communicate with the thermal imaging system. The thermal imaging system includes a thermal imaging module configured to provide thermal image data corresponding to a projected course for a mobile platform. The logic device is configured to receive the thermal image data, receive orientation and/or position data corresponding to the thermal image data, and generate maneuvering obstacle information corresponding to the projected course based, at least in part, on the orientation and/or position data and/or the thermal image data.
1. A system comprising:
a thermal imaging system, wherein the thermal imaging system comprises a thermal imaging module configured to provide thermal image data corresponding to a projected course for a mobile platform;
a visible spectrum imaging system coupled to the mobile platform and configured to provide visible spectrum image data corresponding to the projected course;
an orientation and/or position sensor coupled to the mobile platform and configured to provide orientation and/or position data associated with motion of the mobile platform;
a ranging sensor system coupled to the mobile platform and configured to provide ranging sensor data corresponding to the projected course; and
a logic device configured to communicate with the thermal imaging system, the visible spectrum imaging system, and the orientation and/or position sensor, wherein the logic device is configured to:
receive the thermal image data corresponding to the projected course;
receive visible spectrum image data corresponding to the thermal image data;
receive orientation and/or position data corresponding to the thermal image data;
receive ranging sensor data corresponding to the thermal image data;
generate blended image data based, at least in part, on the received visible spectrum image data and the received thermal image data;
generate maneuvering obstacle information corresponding to the projected course based, at least in part, on the blended image data, the ranging sensor data, and the orientation and/or position data, wherein the logic device is configured to generate the maneuvering obstacle information at least by applying a fused sensor data trained convolution neural network (CNN) to a combination of the ranging sensor data, the visible spectrum image data, and the thermal image data to obtain the maneuvering obstacle information; and
selectively activate a braking system of the mobile platform based, at least in part, on the generated maneuvering obstacle information corresponding to the projected course.
2. The system of claim 1 ,
wherein the logic device is configured to:
generate the maneuvering obstacle information by associating each detected maneuvering object with a map score and determining a range estimate for each detected maneuvering object associated with a map score above a threshold value.
3. The system of claim 1 ,
wherein the logic device is configured to:
generate the maneuvering obstacle information corresponding to the projected course by applying a blended imagery trained convolution neural network (CNN) to the blended image data to obtain the maneuvering obstacle information.
4. The system of claim 1 , wherein the logic device is configured to:
generate the blended image data based on pixel values of the received visible spectrum image data and pixel values of the received thermal image data to obtain the blended image data.
5. The system of claim 4 , wherein the logic device is configured to generate the blended image data by modulating pixel values of the received visible spectrum image data based on overlapping pixel values of the received thermal image data to obtain the blended image data.
6. The system of claim 1 , wherein:
the ranging sensor system comprises a grille mounted radar system and/or a grille mounted lidar system.
7. The system of claim 1 , further comprising a communication module configured to establish a wireless communication link with an update server associated with the mobile platform, wherein:
the thermal imaging system is configured to be coupled to the mobile platform; and
the logic device is configured to receive the thermal image data from the thermal imaging system as the mobile platform maneuvers along the projected course and to report the generated maneuvering obstacle information corresponding to the projected course over the wireless communication link to the update server.
8. The system of claim 1 , wherein the logic device is further configured to adjust a frame rate associated with the thermal imaging system and/or the visible spectrum imaging system based on a speed of the mobile platform.
9. A method comprising:
receiving thermal image data from a thermal imaging system comprising a thermal imaging module configured to provide thermal image data corresponding to a projected course for a mobile platform;
receiving orientation and/or position data corresponding to the thermal image data from an orientation and/or position sensor coupled to the mobile platform;
receiving visible spectrum image data corresponding to the thermal image data from a visible spectrum imaging system coupled to the mobile platform and configured to provide visible spectrum image data corresponding to the projected course;
receiving ranging sensor data corresponding to the thermal image data from a ranging sensor system coupled to the mobile platform and configured to provide ranging sensor data corresponding to the projected course;
generating blended image data based, at least in part, on the received visible spectrum image data and the received thermal image data;
generating maneuvering obstacle information corresponding to the projected course based, at least in part, on the ranging sensor data, the blended image data, and the orientation and/or position data, wherein the generating the maneuvering obstacle information comprises applying a fused sensor data trained convolution neural network (CNN) to a combination of the ranging sensor data, the visible spectrum image data, and the thermal image data to obtain the maneuvering obstacle information; and
selectively activating a braking system of the mobile platform based, at least in part, on the generated maneuvering obstacle information corresponding to the projected course.
10. The method of claim 9 ,
wherein the generating the maneuvering obstacle information comprises associating each detected maneuvering object with a map score and determining a range estimate for each detected maneuvering object associated with a map score above a threshold value.
11. The method of claim 9 ,
wherein the generating the maneuvering obstacle information comprises applying blended imagery trained convolution neural network (CNN) to the blended image data to obtain the maneuvering obstacle information.
12. The method of claim 9
wherein the blended image data is based on pixel values of the received visible spectrum image data and pixel values of the received thermal image data.
13. The method of claim 12 , wherein the generating the maneuvering obstacle information comprises modulating pixel values of the received visible spectrum image data based on overlapping pixel values of the received thermal image data to obtain the blended image data.
14. The method of claim 9 , wherein:
the ranging sensor system comprises a radar system and/or a lidar system.
15. The method of claim 9 , further comprising:
receiving the thermal image data from the thermal imaging system as the mobile platform maneuvers along the projected course; and
reporting the generated maneuvering obstacle information corresponding to the projected course over a wireless communication link to an update server associated with the mobile platform via a communication module configured to establish the wireless communication link with the update server.
16. The system of claim 9 , further comprising adjusting a frame rate associated with the thermal imaging system and/or the visible spectrum imaging system based on a speed of the mobile platform.