Feature detection for automated air-to-air refueling (A3R)
Aspects of the disclosure provide solutions for automated air-to-air refueling (A3R) and assisted air-to-air refueling. Examples include: receiving a video frame; generating, from the video frame, a plurality of images having differing decreasing resolutions; detecting, within each of the plurality of images, a set of aircraft keypoints for an aircraft to be refueled; merging the sets of aircraft keypoints into a set of merged aircraft keypoints; based on at least the merged aircraft keypoints, determining a position of a fuel receptacle on the aircraft; and determining a position of a boom tip of an aerial refueling boom. Some examples include, based on at least the position of the fuel receptacle and the position of the boom tip, controlling the aerial refueling boom to engage the fuel receptacle, and for some examples, the video frame is monocular (e.g., provided by a single camera).
1. A method of air-to-air refueling, the method comprising:
receiving a video frame;
generating, from the video frame, a first plurality of images having differing decreasing resolutions;
detecting, within each of the first plurality of images, a set of aircraft keypoints for an aircraft to be refueled;
merging the sets of aircraft keypoints into a set of merged aircraft keypoints;
determining whether a merged aircraft keypoint requires correction;
based on at least determining that the merged aircraft keypoint requires correction, shifting a position of the merged aircraft keypoint in an aircraft keypoint heatmap;
based on at least the merged aircraft keypoints, determining a position of a fuel receptacle on the aircraft;
cropping the video frame to a video region containing a boom tip of an aerial refueling boom;
generating, from the video region, a second plurality of images having differing decreasing resolutions;
detecting, within each of the second plurality of images, a boom tip keypoint for the boom tip;
merging the boom tip keypoints into a set of merged boom tip keypoints; and
based on the merged boom tip keypoints, determining a position of the boom tip.
2. The method of claim 1 , further comprising:
based on at least the position of the fuel receptacle and the position of the boom tip, controlling the aerial refueling boom to engage the fuel receptacle.
3. The method of claim 1 , wherein the video frame is provided by a single camera.
4. The method of claim 1 , further comprising:
providing, to the aircraft, maneuvering information to facilitate engaging the fuel receptacle with the aerial refueling boom.
5. The method of claim 1 , further comprising:
based on at least the position of the fuel receptacle and the position of the boom tip, determining whether controlling the aerial refueling boom to engage the fuel receptacle is within safety parameters; and
based on at least determining that controlling the aerial refueling boom to engage the fuel receptacle is not within safety parameters, generating an alert.
6. The method of claim 1 , further comprising:
determining whether a merged aircraft keypoint is missing; and
based on at least determining that the merged aircraft keypoint is missing, inserting an additional aircraft keypoint into the aircraft keypoint heatmap.
7. The method of claim 1 , further comprising:
training a first neural network (NN) with a plurality of labeled images of a scene, the plurality of labeled images having differing decreasing resolutions of a common scene;
wherein detecting, within each of the first plurality of images, a set of aircraft keypoints is performed via an aircraft keypoint detector, wherein the aircraft keypoint detector comprises the first neural network; and
wherein detecting, within each of the second plurality of images, a boom tip keypoint is performed via a boom tip keypoint detector, wherein the boom tip keypoint detector comprises a second neural network.
8. The method of claim 1 , further comprising:
performing temporal filtering of aircraft keypoints.
9. The method of claim 1 , wherein determining the position of the fuel receptacle on the aircraft comprises:
fusing proximity sensor measurements of the aircraft with the merged aircraft keypoints.
10. A system for fuel receptacle and boom tip position and pose estimation for aerial refueling, the system comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving a video frame;
generating, from the video frame, a first plurality of images having differing decreasing resolutions;
detecting, within each of the first plurality of images, a set of aircraft keypoints for an aircraft to be refueled;
merging the sets of aircraft keypoints into a set of merged aircraft keypoints;
determining whether a merged aircraft keypoint requires correction;
based on at least determining that the merged aircraft keypoint requires correction, shifting a position of the merged aircraft keypoint in an aircraft keypoint heatmap;
based on at least the merged aircraft keypoints, determining a position of a fuel receptacle on the aircraft;
cropping the video frame to a video region containing a boom tip of an aerial refueling boom;
generating, from the video region, a second plurality of images having differing decreasing resolutions;
detecting, within each of the second plurality of images, a boom tip keypoint for the boom tip;
merging the boom tip keypoints into a set of merged boom tip keypoints; and
based on the merged boom tip keypoints, determining a position of the boom tip.
11. The system of claim 10 , further comprising:
a boom control that controls the aerial refueling boom to engage the fuel receptacle, based on at least the position of the fuel receptacle and the position of the boom tip.
12. The system of claim 10 , further comprising:
a camera, wherein the video frame is provided by the camera; and
a proximity sensor, wherein determining the position of the fuel receptacle on the aircraft comprises:
fusing proximity sensor measurements of the aircraft with the merged aircraft keypoints.
13. The system of claim 10 , wherein the operations further comprise:
providing, to the aircraft, maneuvering information to facilitate engaging the fuel receptacle with the aerial refueling boom.
14. The system of claim 10 , further comprising:
performing temporal filtering of aircraft keypoints.
15. The system of claim 10 , wherein the operations further comprise:
training a first neural network (NN) with a plurality of labeled images of a scene, the plurality of labeled images having differing decreasing resolutions of a common scene;
wherein detecting, within each of the first plurality of images, a set of aircraft keypoints is performed via an aircraft keypoint detector, wherein the aircraft keypoint detector comprises the first neural network; and
wherein detecting, within each of the second plurality of images, a boom tip keypoint is performed via a boom tip keypoint detector, wherein the boom tip keypoint detector comprises a second neural network.
16. A computer program product, comprising a non-transitory computer usable medium having a computer readable program code embodied therein, the computer readable program code adapted to be executed to implement a method of air-to-air refueling, the method comprising:
receiving a video frame;
generating, from the video frame, a first plurality of images having differing decreasing resolutions;
detecting, within each of the first plurality of images, a set of aircraft keypoints for an aircraft to be refueled;
merging the sets of aircraft keypoints into a set of merged aircraft keypoints;
determining whether a merged aircraft keypoint requires correction;
based on at least determining that the merged aircraft keypoint requires correction, shifting a position of the merged aircraft keypoint in an aircraft keypoint heatmap;
based on at least the merged aircraft keypoints, determining a position of a fuel receptacle on the aircraft;
cropping the video frame to a video region containing a boom tip of an aerial refueling boom;
generating, from the video region, a second plurality of images having differing decreasing resolutions;
detecting, within each of the second plurality of images, a boom tip keypoint for the boom tip;
merging the boom tip keypoints into a set of merged boom tip keypoints; and
based on the merged boom tip keypoints, determining a position of the boom tip.
17. The computer program product of claim 16 , wherein the method further comprises:
based on at least the position of the fuel receptacle and the position of the boom tip, controlling the aerial refueling boom to engage the fuel receptacle.
18. The computer program product of claim 16 , wherein the method further comprises:
based on at least the position of the fuel receptacle and the position of the boom tip, determining whether controlling the aerial refueling boom to engage the fuel receptacle is within safety parameters; and
based on at least determining that controlling the aerial refueling boom to engage the fuel receptacle is not within safety parameters, generating an alert.
19. The computer program product of claim 16 , wherein the method further comprises:
training a first neural network (NN) with a plurality of labeled images of a scene, the plurality of labeled images having differing decreasing resolutions of a common scene;
wherein detecting, within each of the first plurality of images, a set of aircraft keypoints is performed via an aircraft keypoint detector, wherein the aircraft keypoint detector comprises the first neural network; and
wherein detecting, within each of the second plurality of images, a boom tip keypoint is performed via a boom tip keypoint detector, wherein the boom tip keypoint detector comprises a second neural network.
20. The computer program product of claim 16 , wherein the method further comprises:
training a neural network (NN) to insert an additional aircraft keypoint into the aircraft keypoint heatmap.