Systems and methods for calculating refueling tanker boom 3D position for aerial refueling
Disclosed herein is methods, systems, and aircraft for performing image analysis for aiding refueling operations. A tanker aircraft includes a camera, a refueling boom, a camera configured to generate a two-dimensional (2D) image of the refueling boom, a processor, and non-transitory computer readable storage media storing code. The code is executable by the processor to perform operations including receiving the two-dimensional (2D) image from the camera, determining 2D keypoints of the refueling boom located within the 2D image based on a predefined point model of the refueling boom, determining a 6 degree-of-freedom (6DOF) pose using the 2D keypoints and the corresponding three-dimensional (3D) space 3D keypoints, optimizing 3D keypoints associated with moveable components of the refueling boom in response to a plurality of boom control parameters to produce optimized 3D keypoints, and estimating a position of a tip of the refueling boom based on the 6DOF pose.
1 . A method comprising:
receiving a two-dimensional (2D) image from a camera of a tanker aircraft;
determining 2D keypoints of a refueling boom located within the 2D image based on a predefined point model of a refueling boom;
determining a 6 degree-of-freedom (6DOF) pose using the determined 2D keypoints and corresponding three-dimensional (3D) space keypoints;
optimizing 3D keypoints associated with moveable components of the refueling boom by iteratively solving for a plurality of boom control parameters for the moveable components, based on a difference between the determined 2D keypoints and projected 2D keypoints generated by projecting the 3D keypoints onto the 2D image using the determined 6DOF pose to produce optimized 3D keypoints such that the projected 2D keypoints substantially align with the determined 2D keypoints; and
estimating a position of the refueling boom based on the 6DOF pose and the optimized 3D keypoints.
2 . The method of claim 1 , wherein determining the 6DOF pose comprises performing a perspective-n-point algorithm using the determined 2D keypoints and the 3D keypoints to produce the 6DOF pose.
3 . The method of claim 1 , further comprising:
receiving a boom tip extension value; and
estimating a position of a tip of the refueling boom based on the boom tip extension value.
4 . The method of claim 1 , wherein the boom control parameters comprise at least one angle between two components of the refueling boom, wherein at least one of the two components comprises a component that is moveable relative to a main body of the refueling boom.
5 . The method of claim 1 , wherein:
generating the projected 2D keypoints is based on the plurality of boom control parameters for the moveable components of the refueling boom;
the method further comprises:
determining the 6DOF pose prior to iteratively solving for the plurality of boom control parameters; and
optimizing the 3D keypoints associated with moveable components of the refueling boom without re-determining the 6DOF pose.
6 . The method of claim 5 , wherein optimizing is performed using an initial value.
7 . The method of claim 1 , further comprising sending the position of the refueling boom to an output device.
8 . The method of claim 7 , wherein the output device comprises an automated refueling system, a pilot director light system, or a boom operator interface.
9 . A tanker aircraft comprising:
a refueling boom;
a camera configured to generate a two-dimensional (2D) image of the refueling boom;
a processor; and
non-transitory computer readable storage media storing code, the code being executable by the processor to perform operations comprising:
receiving the two-dimensional (2D) image from the camera;
determining 2D keypoints of the refueling boom located within the 2D image based on a predefined point model of the refueling boom;
determining a 6 degree-of-freedom (6DOF) pose using the determined 2D keypoints and corresponding three-dimensional (3D) space keypoints;
optimizing 3D keypoints associated with moveable components of the refueling boom by iteratively solving for a plurality of boom control parameters for the moveable components, based on a difference between the determined 2D keypoints and projected 2D keypoints generated by projecting the 3D keypoints onto the 2D image using the determined 6DOF pose to produce optimized 3D keypoints such that the projected 2D keypoints substantially align with the determined 2D keypoints; and
estimating a position of a tip of the refueling boom based on the 6DOF pose and on the optimized 3D keypoints.
10 . The tanker aircraft of claim 9 , further comprising transforming the determined 2D keypoints to 3D space by performing a perspective-n-point algorithm using the determined 2D keypoints and the 3D keypoints to produce the 6DOF pose.
11 . The tanker aircraft of claim 9 , wherein:
the tanker aircraft further comprises a boom tip sensor; and
estimating the position of the tip of the refueling boom comprises:
receiving a boom tip extension value from the boom tip sensor; and
estimating the position of the tip of the refueling boom based on the boom tip extension value.
12 . The tanker aircraft of claim 9 , wherein the boom control parameters comprise at least one angle between two components of the refueling boom, wherein at least one of the two components comprises a component that is moveable relative to a main body of the refueling boom.
13 . The tanker aircraft of claim 12 , wherein optimizing comprises optimizing for a predefined amount of time or until a result is within a predefined range.
14 . The tanker aircraft of claim 13 , wherein optimizing is performed using an initial value.
15 . The tanker aircraft of claim 9 , wherein:
the tanker aircraft further comprises an output device; and
the processor further performs an operation of sending the position of the tip of the refueling boom to the output device.
16 . The tanker aircraft of claim 15 , wherein the output device comprises an automated refueling system, a pilot director light system, or a boom operator interface.
17 . A refueling system comprising:
a processor; and
non-transitory computer readable storage media storing code, the code being executable by the processor to perform operations comprising:
receiving a two-dimensional (2D) image from a camera;
determining 2D keypoints of a refueling boom located within the 2D image based on a predefined point model of the refueling boom;
determining a 6 degree-of-freedom (6DOF) pose using the determined 2D keypoints and corresponding three-dimensional (3D) keypoints;
optimizing 3D keypoints associated with moveable components of the refueling boom by iteratively solving for a plurality of boom control parameters for the moveable components, based on a difference between the determined 2D keypoints and projected 2D keypoints generated by projecting the 3D keypoints onto the 2D image using the determined 6DOF pose to produce optimized 3D keypoints such that the projected 2D keypoints substantially align with the determined 2D keypoints; and
estimating a position of a tip of the refueling boom based on the 6DOF pose and on the optimized 3D keypoints.
18 . The refueling system of claim 17 , further comprising transforming the 2D keypoints to 3D space by performing a perspective-n-point algorithm using the determined 2D keypoints and 3D keypoints to produce the 6DOF pose.
19 . The refueling system of claim 17 , wherein the boom control parameters comprise at least one angle between two components of the refueling boom, wherein at least one of the two components comprises a component that is moveable relative to a main body of the refueling boom.
20 . The refueling system of claim 19 , wherein optimizing comprises optimizing for a predefined amount of time or until a result is within a predefined range responsive to an initial value.