IP Library › Granted Patent US 12,307,911
Granted Patent B2
US 12,307,911 · App. 18/244,361 · Granted May 20, 2025

Multi-sensor data fusion-based aircraft detection, tracking, and docking

Inventors: Rajesh V. Poojary (Bangalore, IN); Raghu Seelamonthula (Bangalore, IN); Vinay Sridhara Murthy (Bangalore, IN); Manuj Sharma (Bangalore, IN)
Assignee: Honeywell International Inc.
G08G5/51B64D45/00G05D1/0202G06F18/251G06T7/521G06V20/13
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,307,911
App. No.
18/244,361
Granted
May 20, 2025
Kind
B2
Abstract

Tracking aircraft in and near a ramp area is described herein. One system for tracking an aircraft, comprising: a sensor array, positioned at or near a ramp area at an airport, having a camera image sensor device and a light detection and ranging (LIDAR) sensor device that are both located at a single sensor array location with respect to the ramp area; the camera image sensor device configured to capture camera image data illustrating the aircraft while the aircraft is approaching or in the ramp area; the LIDAR sensor device configured to capture LIDAR sensor device data illustrating the aircraft while the aircraft is approaching or in the ramp area.

Claims (25)

1. A system for tracking an aircraft, comprising:

a sensor array, positioned at or near a ramp area at an airport, having a camera image sensor device and a light detection and ranging (LIDAR) sensor device that are both located at a single sensor array location with respect to the ramp area;

the camera image sensor device configured to capture camera image data illustrating the aircraft while the aircraft is approaching or in the ramp area;

a computing device at or near the ramp area of the airport having memory and a processor associated therewith configured to receive the camera image data from the camera image sensor device;

the LIDAR sensor device configured to capture LIDAR sensor device data illustrating the aircraft while the aircraft is approaching or in the ramp area; and

the computing device having the memory and the processor associated therewith configured to receive the LIDAR sensor device data, the computing device configured to:

use a combination of the camera image data and the LIDAR sensor device data associated with the aircraft to determine a position of the aircraft approaching or in the ramp area; and

cause a display of messages to provide directional and speed guidance with respect to a center line and a parking location to a pilot based, at least in part, on the position of the aircraft.

2. The system of claim 1 , wherein the computing device is configured to receive the LIDAR sensor device data and merge the camera image data and the LIDAR sensor device data associated with the aircraft into a merged data set to create the combination of the camera image data and the LIDAR sensor device data.

3. The system of claim 2 , wherein the merged data set is used to form a three dimensional virtual model illustrating at least a portion of the aircraft to determine the position of the aircraft approaching or in the ramp area.

4. The system of claim 1 , wherein the camera image data is used to determine an image vision pose estimate including an image plane that is used to orient a three dimensional virtual model of the aircraft with respect to a virtual ramp area.

5. The system of claim 1 , wherein the camera image sensor device is configured to provide data for multiple image frames taken at different times and each image frame is used to determine a plurality of positions of the aircraft at different times.

6. The system of claim 5 , wherein the determined plurality of positions at different times are compared to estimate a speed of the aircraft, and

wherein the computing device is configured to cause the display of the messages based, also at least in part, on the estimated speed of the aircraft.

7. The system of claim 5 , wherein the determined plurality of positions at different times are compared to estimate a direction of the aircraft, and

wherein the computing device is configured to cause the display of the messages based, also at least in part, on the estimated direction of the aircraft.

8. A method for tracking an aircraft, comprising:

providing a sensor array, positioned at or near a ramp area at an airport, having a camera image sensor device and a light detection and ranging (LIDAR) sensor device that are both located at a single sensor array location with respect to the ramp area;

capturing camera image data illustrating an aircraft while the aircraft is in a field of view of the camera image sensor device while the aircraft is approaching or in the ramp area;

capturing LIDAR sensor device data illustrating the aircraft while the aircraft is in a field of view of the LIDAR sensor device while the aircraft is approaching or in the ramp area;

using a combination of the camera image data and the LIDAR sensor device data illustrating the aircraft to determine a position of the aircraft approaching or in the ramp area; and

displaying messages to provide directional and speed guidance with respect to a center line and a parking location to a pilot based, at least in part, on the position of the aircraft.

9. The method of claim 8 , wherein the method comprises merging the camera image data and the LIDAR sensor device data illustrating the aircraft into a merged data set to create the combination of the camera image data and the LIDAR sensor device data.

10. The method of claim 9 , wherein the method comprises forming a three dimensional virtual model using the merged data set, wherein the three dimensional virtual model illustrates at least a portion of the aircraft to determine the position of the aircraft approaching or in the ramp area.

11. The method of claim 8 , wherein the method comprises determining an image vision pose estimate including an image plane that is used to orient a three dimensional virtual model of the aircraft with respect to a virtual ramp area using the camera image data.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 11, 2023
From: POOJARY, RAJESH V.; SEELAMONTHULA, RAGHU; SRIDHARA MURTHY, VINAY; SHARMA, MANUJ
To: HONEYWELL INTERNATIONAL INC.
Reel/Frame 064856/0110 →
Continuity (3)
Continuation 18088937 · Dec 27, 2022
Continuation 16983657 · Aug 3, 2020
Related Publication 20240046807A1 · Feb 8, 2024
References Cited (53)
US 5675661A · Richman · 1997 [cited by applicant]
US 10255520B2 · Deng et al. · 2019 [cited by applicant]
US 10354805B2 · Hemphill · 2019 [cited by applicant]
US 10989791B2 · Tuxen · 2021 [cited by applicant]
US 11538349B2 · Poojary · 2022 [cited by applicant]
US 20130062459A1 · Cox · 2013 [cited by applicant]
US 20130176425A1 · Lener · 2013 [cited by applicant]
US 20150021431A1 · Kracht · 2015 [cited by applicant]
US 20170178520A1 · Papageorgiou · 2017 [cited by examiner]
US 20170226764A1 · Nussbaum · 2017 [cited by applicant]
US 20190310373A1 · Rutkiewicz · 2019 [cited by applicant]
US 20200064483A1 · Li · 2020 [cited by applicant]
US 20200180783A1 · Cox et al. · 2020 [cited by applicant]
US 20220066025A1 · Berkmo · 2022 [cited by applicant]
CN 108932875A · 2018 [cited by applicant]
CN 111427374A · 2020 [cited by applicant]
DE 102014112269 · 2014 [cited by applicant]
EP 3680689 · 2020 [cited by applicant]
EP 3680689A1 · 2020 [cited by applicant]
JP 2001187599A · 2001 [cited by applicant]
JP 4071412 · 2008 [cited by applicant]
KR 101314566B1 · 2013 [cited by applicant]
WO 2015005534A1 · 2015 [cited by applicant]
WO 2017162432 · 2017 [cited by applicant]
Chinrungrueng J, “Smart Parking: An Application of Optical Wireless Sensor Network,” International Symposium on Applications and the Internet Workshops, Jan. 15, 2007, 5 pgs. [cited by applicant]
Tran, V.L., “A Study on Ship Automatic Berthing with Assistance of Auxiliary Devices,” International Journal of Naval Architecture and Ocean Engineering, 2012, 13 pgs. [cited by applicant]
Mi, C., “Ship Identification Algorithm based on 3D Point Cloud for Automated Ship Loaders.” Journal of Coastal Research, (73), 2015, 9 pgs. [cited by applicant]
Thupakula, K., “Data Fusion Techniques for Object Identification in Airport Environment.” SAE Technical Paper 2017-01-2109, Sep. 19, 2017, 6 pgs. [cited by applicant]
Miller, I., “Team Cornell's Skynet: Robust Perception and Planning in an Urban Environment,” Journal of Field Robotics, 25(8), 2008, 36 pgs. [cited by applicant]
Schumann, O., “Semantic Segmentation on Radar Point Clouds,” International Conference on Information Fusion (FUSION), Jul. 2018, 9 pgs. [cited by applicant]
Aldoma, A., “Tutorial: Point Cloud Library: Three-Dimensional Object Recognition and 6 DoF Pose Estimation,” IEEE Robotics & Automation Magazine 19, No. 3, 2012, 13 pgs. [cited by applicant]
I. Miller, “Sensitivity Analysis of a Tightly-Coupled GPS/INS System for Autonomous Navigation,” IEEE Transactions on Aerospace and Electronic Systems, 48(2), 2012, 22 pgs. [cited by applicant]
Mund, J., “Introducing Lidar Point Cloud-based Object Classification for Safer Apron Operations,” International Symposium on Enhanced Solutions for Aircraft and Vehicle Surveillance Applications, Apr. 2016, 11 pgs. [cited by applicant]
I. Miller, “Efficient Unbiased Tracking of Multiple Dy-namic Obstacles Under Large Viewpoint Changes,” IEEE Transactions on Robotics, 27(1), Feb. 2011, 19 pgs. [cited by applicant]
F. Havlak, “Discrete and Continuous, Probabilistic Anticipation for Autonomous Robots in Urban Environments,” IEEE Transactions on Robotics, 30(2), Apr. 2014, 15 pgs. [cited by applicant]
I. Miller, “Map-aided Localization in Sparse Global Positioning System Environments using Vision and Particle Filtering,” Journal of Field Robotics, 28(5), Sep. 2011, 26 pgs. [cited by applicant]
Campbell, M. E., “Distributed Data Fusion: Neighbors, Rumors, and the Art of Collective Knowledge,” IEEE Control Systems, 36(4), Aug. 2016, 28 pgs. [cited by applicant]
Deng, J., “Imagenet: A large-scale Hierarchical Image Database,” IEEEConference on Computer Vision and Pattern Recognition, Jun. 2009, 9 pgs. [cited by applicant]
Rusu, R.B., “3D is Here: Point Cloud Library (PCL)” IEEE International Conference on Robotics and Automation (ICRA), 2011, 5 pgs. [cited by applicant]
Ip, C.Y., “Retrieving Matching CAD Models by using Partial 3D Point Clouds,” Computer-Aided Design and Applications, 4(5), 2007, 11 pgs. [cited by applicant]
Wohlkinger, W., “3Dnet: Large-scale Object Class Recognition from CAD Models,” IEEE International Conference on Robotics and Automation, May 2012, 9 phs. [cited by applicant]
Vock, R., “Fast Template Matching and Pose Estimation in 3D Point Clouds,” Computers & Graphics, 79, 2019, 11 pgs. [cited by applicant]
Qi, C.R., “Pointnet: Deep Learning on Point Sets for 3D Classification and Segmentation,” IEEE Conference on Computer Vision and Pattern Recognition, 2017, 20 pgs. [cited by applicant]
Wang, Y., “Pseudo-Lidar from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving,” IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, 10pgs. [cited by applicant]
Bar-Shalom, “Y. Estimation with applications to tracking and navigation: theory algorithms and software.” John Wiley & Sons, 2001, 100 pgs. [cited by applicant]
Petrovskaya, A. “Model based vehicle detection and tracking for autonomous urban driving.” Autonomous Robots, 26 (2-3), 2009, pp. 123-139. [cited by applicant]
Campbell, M. “Expert Declaration”, retrieved Oct. 2023, 101 pgs. [cited by applicant]
Berkmo, A. “Annoted version of Berkmo highlighting paragraphs included in Berkmo but absent from EP'689” Mar. 2022, 23 pgs. [cited by applicant]
Weinstein, M. “Post Grant Review” retrieved Oct. 2023, 92 pgs. [cited by applicant]
Renuka, et al. “Smart Ramp Information Display System”, International Journal of Pure and Applied Mathematics, vol. 119, No. 16, 2018, 6 pages. [cited by applicant]
Rodriguez-Garavito, et al., “Automatic Laser And Camera Extrinsic Calibration for Data Fusion Using Road Plane”, IEEE Intelligent Transportation Systems Conference, 2006, 6 pages. [cited by applicant]
Extended European Search Report for related EP Application No. 21189230.2, mailed Dec. 17, 2021 (11 pgs). [cited by applicant]
[cited by applicant]