Fuel receptacle and boom tip position and pose estimation for aerial refueling

1. A method of fuel receptacle and boom tip position and pose estimation for aerial refueling, the method comprising:

receiving a video frame;

determining via a first pipeline, within the video frame, a plurality of aircraft keypoints for an aircraft to be refueled, wherein the first pipeline performs operations including:

generating an aircraft keypoint heatmap based on the plurality of aircraft keypoints in a first region of the video frame containing the aircraft to be refueled, wherein each of the plurality of aircraft keypoints corresponds to a region of pixels in the aircraft keypoint heatmap having values dropping, according to a probability density function, with increasing distance from the center of the region of pixels; and

applying a first threshold to the aircraft keypoint heatmap to eliminate aircraft keypoints having a low confidence level;

based on at least the plurality of aircraft keypoints, determining via the first pipeline a position and pose of a fuel receptacle on the aircraft;

determining via a second pipeline a position and pose of a boom tip of an aerial refueling boom, wherein the second pipeline operates in parallel with the first pipeline, and wherein the second pipeline performs operations including:

generating a boom tip keypoint heatmap based on one or more boom tip keypoints in a second region of the video frame containing the boom tip, wherein each of the one or more boom tip keypoints corresponds to a region of pixels in the boom tip keypoint heatmap having values dropping, according to a probability density function, with increasing distance from the center of the region of pixels; and

applying a second threshold to the boom tip keypoint heatmap to eliminate boom tip keypoints having a low confidence level; and

based on at least the position and pose of the fuel receptacle and the position and pose of the boom tip, controlling the aerial refueling boom to engage the fuel receptacle;

generating a second video frame, wherein generating the second video frame comprises one or more of;

based on at least the position and pose of the aircraft, overlaying an aircraft model projection in the first video frame; or

based on at least the position and pose of the boom tip, overlaying a boom model projection in the first video frame: and

displaying the second videoframe with one or more of the overlay of the aircraft model projection or the overlay of the boom model projection.

2. The method of claim 1 , wherein the video frame is provided by a single camera.

3. The method of claim 1 , further comprising:

determining via the first pipeline, within the video frame, an aircraft bounding box for the aircraft to be refueled using a neural network (NN), the NN comprising a convolutional neural network (CNN); and

cropping the video frame to the aircraft bounding box, wherein determining the aircraft keypoints comprises determining the aircraft keypoints within the aircraft bounding box.

4. The method of claim 1 , wherein generating the aircraft keypoint heatmap comprises:

using a convolutional neural network (CNN).

5. The method of claim 1 , wherein determining the position and pose of the boom tip comprises:

determining via the second pipeline, within the video frame, a boom tip bounding box for the boom tip using boom control parameters and camera calibration information; and

determining, within the boom tip bounding box, a boom tip keypoint for the boom tip.

6. The method of claim 1 , further comprising:

displaying the video frame with one or more of an overlay of an aircraft model projection or an overlay of a boom model projection.

7. A system for fuel receptacle and boom tip position and pose estimation for aerial refueling, the system comprising:

one or more processors; and

a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving a video frame;

determining via a first pipeline, within the video frame, a plurality of aircraft keypoints for an aircraft to be refueled, wherein the first pipeline performs operations including:

generating an aircraft keypoint heatmap based on the plurality of aircraft keypoints in a first region in the video frame containing the aircraft to be refueled, wherein each of the plurality of aircraft keypoints corresponds to a region of pixels in the aircraft keypoint heatmap having values dropping, according to a probability density function, with increasing distance from the center of the region of pixels; and

applying a first threshold to the aircraft keypoint heatmap to eliminate aircraft keypoints having a low confidence level;

based on at least the plurality of aircraft keypoints, determining via the first pipeline a position and pose of a fuel receptacle on the aircraft;

determining via a second pipeline a position and pose of a boom tip of an aerial refueling boom, wherein the second pipeline operates in parallel with the first pipeline, and wherein the second pipeline performs operations including:

generating a boom tip keypoint heatmap based on one or more boom tip keypoints in a second region in the video frame containing the boom tip, wherein each of the one or more boom tip keypoints corresponds to a region of pixels in the boom tip keypoint heatmap having values dropping, according to a probability density function, with increasing distance from the center of the region of pixels; and

applying a second threshold to the boom tip keypoint heatmap to eliminate boom tip keypoints having a low confidence level; and

based on at least the position and pose of the fuel receptacle and the position and pose of

generating a second video frame, whereingenerating the second videoframe comprises one or more of:

based on at least the position and pose of the aircraft, overlaying an aircraft model projection in the first video frame; or

based on at least the position and pose of the boom tip, overlaying a boom model projection in the first video frame; and

displaying the second videoframe with one or more of the overlay of the aircraft model projection or the overlay of the boom model projection.

8. The system of claim 7 , wherein the operations further comprise:

determining via the first pipeline, within the video frame, an aircraft bounding box for the aircraft to be refueled using a neural network (NN), the NN comprising a convolutional neural network (CNN); and

cropping the video frame to the aircraft bounding box, wherein determining the aircraft keypoints comprises determining the aircraft keypoints within the aircraft bounding box.

9. The system of claim 7 , wherein generating the aircraft keypoint heatmap comprises:

using a convolutional neural network (CNN).

10. The system of claim 7 , wherein determining the position and pose of the fuel receptacle comprises:

performing a two-dimensional (2D) to three-dimensional (3D) transform for the aircraft keypoints using a perspective-n-point (PnP) algorithm that utilizes correspondences between the 2D locations of pixels corresponding to the aircraft keypoints and the 3D locations of keypoints in an object model of the aircraft, to position the object model such that the view of the 2D locations matches the view of the 3D locations of keypoints, and to determine the position and pose of the aircraft, and

determining the position and pose of the fuel receptacle based on the position and pose of the aircraft.

11. The system of claim 7 , wherein determining the position and pose of the boom tip comprises:

determining via the second pipeline, within the video frame, a boom tip bounding box for the boom tip using boom control parameters and camera calibration information; and

determining, within the boom tip bounding box, a boom tip keypoint for the boom tip.

12. The system of claim 7 , wherein the operations further comprise:

displaying the video frame with one or more of an overlay of an aircraft model projection or an overlay of a boom model projection.

13. A computer program product, comprising a non-transitory computer usable medium having a computer readable program code embodied therein, the computer readable program code adapted to be executed to implement a method of fuel receptacle and boom tip position and pose estimation for aerial refueling, the method comprising:

receiving a first video frame;

determining via a first pipeline, within the first video frame, a plurality of aircraft keypoints for an aircraft to be refueled, wherein the first pipeline performs operations including:

generating an aircraft keypoint heatmap based on the plurality of aircraft keypoints in a first region of the first video frame containing the aircraft to be refueled, wherein each of the plurality of aircraft keypoints corresponds to a region of pixels in the aircraft keypoint heatmap having values dropping, according to a probability density function, with increasing distance from the center of the region of pixels; and

applying a first threshold to the aircraft keypoint heatmap to eliminate aircraft keypoints having a low confidence level;

based on at least the plurality of aircraft keypoints, determining via the first pipeline a position and pose of a fuel receptacle on the aircraft;

determining via a second pipeline a position and pose of a boom tip of an aerial refueling boom, wherein the second pipeline operates in parallel with the first pipeline, and wherein the second pipeline performs operations including:

generating a boom tip keypoint heatmap based on one or more boom tip keypoints in a second region of the first video frame containing the boom tip, wherein each of the one or more boom tip keypoints corresponds to a region of pixels in the boom tip keypoint heatmap having values dropping, according to a probability density function, with increasing distance from the center of the region of pixels; and

applying a second threshold to the boom tip keypoint heatmap to eliminate boom tip keypoints having a low confidence level;

based on at least the position and pose of the fuel receptacle and the position and pose of the boom tip, controlling the aerial refueling boom to engage the fuel receptacle;

generating a second video frame, wherein generating the second video frame comprises one or more of:

based on at least the position and pose of the aircraft, overlaying an aircraft model projection in the first video frame; or based on at least the position and pose of the boom tip, overlaying a boom model projection in the first video frame; and

displaying the second video frame with one or more of the overlay of the aircraft model projection or the overlay of the boom model projection.

14. The computer program product of claim 13 , wherein the first video frame is provided by a single camera.

15. The computer program product of claim 13 , wherein the method further comprises:

determining via the first pipeline, within the first video frame, an aircraft bounding box for the aircraft to be refueled using a neural network (NN), the NN comprising a convolutional neural network (CNN); and

cropping the first video frame to the aircraft bounding box, wherein determining the aircraft keypoints comprises determining the aircraft keypoints within the aircraft bounding box.

16. The computer program product of claim 13 , wherein generating the aircraft keypoint heatmap comprises:

using a convolutional neural network (CNN).

17. The computer program product of claim 13 , wherein determining the position and pose of the fuel receptacle comprises:

performing a two-dimensional (2D) to three-dimensional (3D) transform for the aircraft keypoints using a perspective-n-point (PnP) algorithm that utilizes correspondences between the 2D locations of pixels corresponding to the aircraft keypoints and the 3D locations of keypoints in an object model of the aircraft, to position the object model such that the view of the 2D locations matches the view of the 3D locations of keypoints, and to determine the position and pose of the aircraft, and

determining the position and pose of the fuel receptacle based on the position and pose of the aircraft.

18. The computer program product of claim 13 , wherein determining the position and pose of the boom tip comprises:

determining via the second pipeline, within the first video frame, a boom tip bounding box for the boom tip using boom control parameters and camera calibration information; and

determining, within the boom tip bounding box, a boom tip keypoint for the boom tip.

19. The method of claim 1 , further comprising:

determining via the second pipeline, within the video frame, a boom tip bounding box for the boom tip using a neural network (NN), the NN comprising a convolutional neural network (CNN); and

cropping the video frame to the boom tip bounding box.

20. The system of claim 7 , wherein the operations further comprise:

determining via the second pipeline, within the video frame, a boom tip bounding box for the boom tip using a neural network (NN), the NN comprising a convolutional neural network (CNN); and

cropping the video frame to the boom tip bounding box.