Multi-sensor data fusion-based aircraft detection, tracking, and docking
Tracking aircraft in and near a ramp area is described herein. One method includes receiving camera image data of an aircraft while the aircraft is approaching or in the ramp area, receiving LIDAR/Radar sensor data of an aircraft while the aircraft is approaching or in the ramp area, merging the camera image data and the LIDAR/Radar sensor data into a merged data set, and wherein the merged data set includes at least one of: data for determining the position and orientation of the aircraft relative to the position and orientation of the ramp area, data for determining speed of the aircraft, data for determining direction of the aircraft, data for determining proximity of the aircraft to a particular object within the ramp area, and data for forming a three dimensional virtual model of at least a portion of the aircraft from the merged data.
1. A system for determining parameters of an aircraft, comprising:
a camera image sensor device configured to capture camera image data of an aircraft at a ramp area of an airport;
a light detection and ranging (LIDAR) sensor device configured to capture LIDAR sensor data of the aircraft at the ramp area of the airport; and
a controller having a processor configured to:
receive the camera image data of the aircraft from the camera image sensor device;
process the received camera image data using machine learning, wherein processing the received camera image data using machine learning includes:
generating a three dimensional virtual model of the aircraft using the received camera image data; and
comparing the generated three dimensional virtual model of the aircraft to a number of aircraft reference models stored in a three dimensional model database to find a match between the generated three dimensional virtual model of the aircraft and one of the number of aircraft reference models;
receive the LIDAR sensor data of the aircraft from the LIDAR sensor device;
merge the processed camera image data and the received LIDAR sensor data into a merged data set;
determine, using the merged data set:
a type of the aircraft;
a speed of the aircraft;
distance data associated with the aircraft;
directional data associated with the aircraft; and
a position and orientation of the aircraft; and
cause a display of a message to provide directional and speed guidance to a pilot based, at least in part, on the determined type of the aircraft, the determined speed of the aircraft, the determined distance data associated with the aircraft, the determined directional data associated with the aircraft, and the determined position and orientation of the aircraft; and
a display device configured to display the message.
2. The system of claim 1 , wherein the controller is configured to:
receive, from an automatic dependent service-broadcast (ADS-B) system, ADS-B data of the aircraft at the ramp area of the airport; and
merge the received ADS-B data into the merged data set.
3. The system of claim 2 , wherein the ADS-B data of the aircraft includes position data and velocity data of the aircraft.
4. The system of claim 2 , wherein the ADS-B data is received from the ADS-B system prior to the aircraft entering a field of view of the camera image sensor device.
5. The system of claim 2 , wherein the ADS-B data is received from the ADS-B system while the aircraft enters a field of view of the camera image sensor device.
6. The system of claim 1 , wherein the controller is configured to:
mask out, using the merged data set, a portion of the ramp area of the airport occupied by the aircraft; and
detect, using the received camera image data, an obstacle in the masked out portion of the ramp area.
7. The system of claim 1 , wherein the controller is configured to store the generated three dimensional virtual model of the aircraft in the three dimensional model database.
8. The system of claim 7 , wherein:
the camera image sensor device is configured to capture camera image data of an additional aircraft at an additional ramp area of the airport; and
the controller is configured to:
receive the camera image data of the additional aircraft from the camera image sensor device; and
process the received camera image data of the additional aircraft using machine learning, wherein processing the received camera image data of the additional aircraft using machine learning includes:
generating a three dimensional virtual model of the additional aircraft using the received camera image data of the additional aircraft; and
comparing the generated three dimensional virtual model of the additional aircraft to the number of aircraft reference models stored in the three dimensional model database, including the generated three dimensional virtual model of the aircraft, to find a match between the generated three dimensional virtual model of the additional aircraft and one of the number of aircraft reference models.
9. The system of claim 1 , wherein the camera image sensor device and the LIDAR sensor device are both located at a same location.
10. The system of claim 1 , wherein the distance data associated with the aircraft comprises:
a distance of the aircraft to a stop point of the aircraft; and
a distance of the aircraft to a gate of the airport.