IP Library › Granted Patent US 12,072,204
Granted Patent B2
US 12,072,204 · App. 17/464,514 · Granted Aug 27, 2024

Landing zone evaluation

Inventors: Yi Dong (Cambridge, MA); Mustafa Hasekioglu (Boston, MA)
Assignee: The Boeing Company
G01C23/00G01C21/005G01C21/20G06T17/05G06V10/34G06V20/17G06V20/58H04N23/90H04N23/51
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,072,204
App. No.
17/464,514
Granted
Aug 27, 2024
Kind
B2
Abstract

Disclosed herein is a system that comprises one or more imaging sensors coupled to an aircraft and one or more navigation sensors coupled to the aircraft proximate to the one or more imaging sensors. The system further comprises an imaging module configured to capture images of a landing zone for the aircraft while the aircraft is in flight. The system additionally comprises a location determining module configured to associate a corresponding one of a plurality of location data with each one of the captured images. The plurality of location data is determined using one or more navigation sensors coupled to the aircraft. The system also comprises an object-identifying module configured to identify and locate one or more objects within the landing zone, relative to the aircraft, using the captured images and the location data associated with the captured images.

Claims (42)

1. A system, comprising:

one or more imaging sensors coupled to an aircraft;

one or more navigation sensors coupled to the aircraft proximate to the one or more imaging sensors;

an imaging module configured to capture images of a landing zone for the aircraft while the aircraft is in flight, wherein the captured images are captured using the one or more imaging sensors;

a location determining module configured to associate a corresponding one of a plurality of location data with each one of the captured images, wherein the plurality of location data is determined using the one or more navigation sensors coupled to the aircraft; and

an object-identifying module configured to apply a point-cloud transformation to the captured images to generate a point cloud by converting image data from a first coordinate frame of the one or more imaging sensors to a second coordinate frame of the aircraft, and to identify and locate one or more objects within the landing zone, relative to the aircraft, using the captured images, the location data associated with the captured images, and the point cloud.

2. The system of claim 1 , wherein each one of the one or more imaging sensors is coupled to the aircraft adjacent to a corresponding navigation sensor of the one or more navigation sensors.

3. The system of claim 2 , wherein an imaging sensor of the one or more imaging sensors and its corresponding navigation sensor are located adjacent to one another within a housing, the housing coupled to an underside of the aircraft.

4. The system of claim 3 , wherein the imaging sensor and its corresponding navigation sensor are located no more than 0.5 meters apart from one another.

5. The system of claim 1 , wherein the aircraft is an autonomous aircraft that processes one or more navigation instructions without user interaction to avoid the one or more objects within the landing zone.

6. The system of claim 1 , wherein the aircraft comprises at least a first imaging sensor oriented along a horizontal plane and a second imaging sensor oriented along a vertical plane.

7. The system of claim 1 , wherein a number and an orientation of the one or more imaging sensors is determined based on one or more specifications of the aircraft, the one or more specifications comprising at least one of a type of the aircraft, a size of the aircraft, a shape of the aircraft, or a configuration of a landing gear for the aircraft.

8. A method, comprising:

capturing images of a landing zone for an aircraft while the aircraft is in flight, wherein the captured images are captured using one or more imaging sensors coupled to the aircraft;

associating a corresponding one of a plurality of location data with each one of the captured images, wherein the plurality of location data is determined using one or more navigation sensors coupled to the aircraft;

applying a point-cloud transformation to the captured images to generate a point cloud by converting image data from a first coordinate frame of the one or more imaging sensors to a second coordinate frame of the aircraft; and

identifying and locating one or more objects within the landing zone, relative to the aircraft, using the captured images, the location data associated with the captured images, and the point cloud.

9. The method of claim 8 , further comprising:

filtering the generated point cloud to generate a cleaned point cloud by identifying and removing statistical outlier data from the point cloud; and

down sampling the cleaned point cloud.

10. The method of claim 9 , further comprising localizing the cleaned point cloud into world coordinates using a temporally aligned mapping of the location data with each of the captured images, each point in the cleaned point cloud associated with latitude, longitude, and elevation values derived from the location data, wherein the localizing generates a localized point cloud.

11. The method of claim 10 , further comprising generating a surface model of the landing zone based on the localized point cloud, the surface model comprising a surface terrain of the landing zone, wherein one or more navigation instructions are derived from the surface model.

12. The method of claim 8 , further comprising providing one or more navigation instructions for avoiding the one or more objects within the landing zone while the aircraft is landing.

13. A computer program product comprising a computer-readable storage medium having program code embodied therein, the program code executable by a processor for:

receiving, from one or more imaging sensors coupled to an aircraft, images of a landing zone for the aircraft, wherein the images have been captured using the one or more imaging sensors while the aircraft is in flight;

associating a corresponding one of a plurality of location data with each one of the captured images, wherein the plurality of location data is determined using one or more navigation sensors coupled to the aircraft;

applying a point-cloud transformation to the captured images to generate a point cloud by converting image data from a first coordinate frame of the one or more imaging sensors to a second coordinate frame of the aircraft; and

identifying and locating one or more objects within the landing zone, relative to the aircraft, using the captured images, the location data associated with the captured images, and the point cloud.

14. The computer program product of claim 13 , wherein the program code is further executable by the processor for:

filtering the generated point cloud to generate a cleaned point cloud by identifying and removing statistical outlier data from the point cloud; and

down sampling the cleaned point cloud.

15. The computer program product of claim 14 , wherein:

the program code is further executable by the processor for localizing the cleaned point cloud into world coordinates using a temporally-aligned mapping of the plurality of location data with the captured images, wherein the localizing generates a localized point cloud; and

each point in the cleaned point cloud is associated with latitude, longitude, and elevation values derived from a corresponding one of the plurality of location data.

16. The computer program product of claim 15 , wherein the program code is further executable by the processor for generating a surface model of the landing zone based on the localized point cloud, wherein the surface model comprises a surface terrain model of the landing zone, and one or more navigation instructions are derived from the surface model.

17. The computer program product of claim 16 , wherein the program code is further executable by the processor for providing a visual graphic of the surface model of the landing zone, the visual graphic highlighting the one or more identified objects within the landing zone such that the one or more identified objects are distinguishable from areas of the landing zone that are free of objects.

18. The computer program product of claim 16 , wherein the program code is further executable by the processor for executing a surface terrain analysis for further surface terrain processing of the landing zone.

19. The computer program product of claim 18 , wherein the surface terrain analysis comprises one or more of:

a plane fitting for determining above-ground objects within the landing zone by fitting one or more planes to the localized point cloud using random sample consensus (“RANSAC”);

a surface roughness for computing a surface roughness index (“SRI”) of the landing zone to highlight corners and edges within the landing zone based on the localized point cloud; and

a surface normal for computing a surface normal for each point of the localized point cloud of the landing zone using singular value decomposition (“SVD”).

20. The computer program product of claim 13 , wherein the program code is further executable by the processor for determining and providing one or more navigation instructions for avoiding the one or more objects within the landing zone while the aircraft is landing.

Assignments (3)
CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE NAME PREVIOUSLY RECORDED AT REEL: 057362 FRAME: 0327. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Sep 9, 2021
From: DONG, YI; HASEKIOGLU, MUSTAFA
To: AURORA FLIGHT SCIENCES CORPORATION, A SUBSIDIARY OF THE BOEING COMPANY
Reel/Frame 057453/0661 →
CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE NAME PREVIOUSLY RECORDED AT REEL: 057362 FRAME: 0327. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Sep 9, 2021
From: DONG, YI; HASEKIOGLU, MUSTAFA
To: AURORA FLIGHT SCIENCES CORPORATION, A SUBSIDIARY OF THE BOEING COMPANY
Reel/Frame 057453/0780 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 1, 2021
From: DONG, YI; HASEKIOGLU, MUSTAFA
To: THE BOEING COMPANY
Reel/Frame 057362/0327 →
Continuity (2)
Provisional Application 63093766 · Oct 19, 2020
Related Publication 20220136860A1 · May 5, 2022