Granted Patent B2

US 12,459,666 · App. 18/118,452 · Granted Nov 4, 2025

Inventors: Maxime Marie Christophe Gariel (San Francisco, CA); Alexander Amin Hamid Bridi (Reston, VA); Robert Eugene Johnston Timpe (Burlingame, CA)

Assignee: Joby Aero, Inc.

B64D45/08G06T7/215G06T7/246G06V10/774G06V20/17G06V20/647G06T2207/10036

Abstract

Claims (81)

1 . A computer-implemented method, comprising:

capturing two-dimensional (2D) image data via one or more cameras of an aircraft;

identifying a portrayal in image data of at least one side stripe of an aircraft runway at a geographical location;

predicting a center point of the 2D image data;

determining respective placement of edges in the 2D image data of a bounding box based on the predicted center point;

applying a three-dimensional (3D) map of the geographic location to the portrayal of the at least one side stripe in the image data;

based on applying the 3D map, determining a current position of the aircraft in the 3D map with respect to the aircraft runway at the geographical location; and

generating bounding box output data that includes the placement of edges in the 2D image data.

2 . The computer-implemented method of claim 1 , wherein determining a current position of an aircraft in the 3D map with respect to the aircraft runway at the geographical location comprises:

determining the aircraft's orientation in the 3D map relative to a known geographic location of the at least one side stripe identified as being portrayed in the image data.

3 . The computer-implemented method of claim 1 , further comprising:

utilizing the determined current position of the aircraft in the 3D map for generating autonomous aircraft data for autonomous control of landing the aircraft on a physical runway that includes a physical instance of the at least one side stripe identified as being portrayed in the image data.

4 . The computer-implemented method of claim 1 , wherein identifying a portrayal in image data of at least one side stripe of an aircraft runway comprises:

generating the bounding box from the 2D image data, the bounding box including the portrayal of the at least one side stripe of the aircraft runway.

5 . The computer-implemented method of claim 4 , wherein generating a bounding box from the image data comprises:

feeding the 2D image data into a neural network, the neural network trained according to image training data, the image training data comprising respective different types of runway images; and

receiving neural network output predicting the center point of the 2D image data.

6 . The computer-implemented method of claim 5 , wherein the respective different types of runway images comprise at least one or more of:

i. images of different runways;

ii. images of runways in different visibility conditions;

iii. images of runways from different altitude perspectives; and

iv. infrared images of runway.

7 . The computer-implemented method of claim 5 , further comprising:

generating normalized image output by applying histogram equalization to the bounding box output data; and

generating masked image output by applying local thresholding to each pixel in each row of pixels in the normalized image output.

8 . The computer-implemented method of claim 7 , further comprising:

generating skeletonized image output by applying image thinning to the masked image output;

generating filtered image output by applying image filtering to the skeletonized image output; and

generating segmented image output by applying segment/clustering extraction to the filtered image output.

9 . A system comprising one or more processors, and a non-transitory computer-readable medium including one or more sequences of instructions that, when executed by the one or more processors, cause the system to perform operations comprising:

capturing two-dimensional (2D) image data via one or more cameras of an aircraft;

identifying a portrayal in image data of at least one side stripe of an aircraft runway at a geographical location;

predicting a center point of the 2D image data;

determining respective placement of edges in the 2D image data of a bounding box based on the predicted center point;

applying a three-dimensional (3D) map of the geographic location to the portrayal of the at least one side stripe in the image data;

based on applying the 3D map, determining a current position of the aircraft in the 3D map with respect to the aircraft runway at the geographical location; and

generating bounding box output data that includes the placement of edges in the 2D image data.

10 . The system of claim 9 , wherein determining a current position of an aircraft in the 3D map with respect to the aircraft runway at the geographical location comprises:

determining the aircraft's orientation in the 3D map relative to a known geographic location of the at least one side stripe identified as being portrayed in the image data.

11 . The system of claim 9 , further comprising:

utilizing the determined current position of the aircraft in the 3D map for generating autonomous aircraft data for autonomous control of landing the aircraft on a physical runway that includes a physical instance of the at least one side stripe identified as being portrayed in the image data.

12 . The system of claim 9 , wherein identifying a portrayal in image data of at least one side stripe of an aircraft runway comprises:

generating the bounding box from the 2D image data, the bounding box including the portrayal of the at least one side stripe of the aircraft runway.

13 . The system of claim 12 , wherein generating a bounding box from the image data comprises:

feeding the 2D image data into a neural network, the neural network trained according to image training data, the image training data comprising respective different types of runway images; and

receiving neural network output predicting a center point of the 2D image data.

14 . The system of claim 13 , wherein the respective different types of runway images comprise at least one or more of:

i. images of different runways;

ii. images of runways in different visibility conditions;

iii. images of runways from different altitude perspectives; and

iv. infrared images of runway.

15 . The system of claim 13 , further comprising:

generating normalized image output by applying histogram equalization to the bounding box output data; and

generating masked image output by applying local thresholding to each pixel in each row of pixels in the normalized image output.

16 . The system of claim 15 , further comprising:

generating skeletonized image output by applying image thinning to the masked image output;

generating filtered image output by applying image filtering to the skeletonized image output; and

generating segmented image output by applying segment/clustering extraction to the filtered image output.

17 . A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:

capturing two-dimensional (2D) image data via one or more cameras of an aircraft;

identifying a portrayal in image data of at least one side stripe of an aircraft runway at a geographical location;

predicting a center point of the 2D image data;

determining respective placement of edges in the 2D image data of a bounding box based on the predicted center point;

applying a three-dimensional (3D) map of the geographic location to the portrayal of the at least one side stripe in the image data;

based on applying the 3D map, determining a current position of the aircraft in the 3D map with respect to the aircraft runway at the geographical location; and

generating bounding box output data that includes the placement of edges in the 2D image data.

18 . The computer program product of claim 17 , wherein determining a current position of an aircraft in the 3D map with respect to the aircraft runway at the geographical location comprises:

determining the aircraft's orientation in the 3D map relative to a known geographic location of the at least one side stripe identified as being portrayed in the image data.

19 . The computer program product of claim 17 , further comprising:

utilizing the determined current position of the aircraft in the 3D map for generating autonomous aircraft data for autonomous control of landing the aircraft on a physical runway that includes a physical instance of the at least one side stripe identified as being portrayed in the image data.

20 . The computer program product of claim 17 , wherein identifying a portrayal in image data of at least one side stripe of an aircraft runway comprises:

capturing two-dimensional (2D) image data via one or more cameras of the aircraft; and

generating the bounding box from the 2D image data, the bounding box including the portrayal of the at least one side stripe of the aircraft runway;

feeding the 2D image data into a neural network, the neural network trained according to image training data, the image training data comprising respective different types of runway images, wherein the respective different types of runway images comprise at least one or more of:

images of different runways, images of runways in different visibility conditions, images of runways from different altitude perspectives and infrared images of runway;

receiving neural network output predicting a center point of the 2D image data;

generating normalized image output by applying histogram equalization to the bounding box output data;

generating masked image output by applying local thresholding to each pixel in each row of pixels in the normalized image output;

generating skeletonized image output by applying image thinning to the masked image output;

generating filtered image output by applying image filtering to the skeletonized image output; and

generating segmented image output by applying segment/clustering extraction to the filtered image output.

Assignments (2)

Continuity (1)

References Cited (99)