Pose estimation refinement for aerial refueling
Aspects of the disclosure provide fuel receptacle position/pose estimation for aerial refueling (derived from aircraft position and pose estimation). A video frame, showing an aircraft to be refueled, is received from a single camera. An initial position/pose estimate is determined for the aircraft, which is used to generating an initial rendering of an aircraft model. The video frame and the initial rendering are used to determining refinement parameters (e.g., a translation refinement and a rotational refinement) for the initial position/pose estimate, providing a refined position/pose estimate for the aircraft. The position/pose of a fuel receptacle on the aircraft is determined, based on the refined position/pose estimate for the aircraft, and an aerial refueling boom may be controlled to engage the fuel receptacle. Examples extract features from the aircraft in the video frame and the aircraft model rendering, and use a deep learning neural network (NN) to determine the refinement parameters.
1. A method of aerial refueling, the method comprising:
receiving a video frame showing an aircraft to be refueled;
receiving an initial position estimate and an initial pose estimate for the aircraft;
based on at least the initial position estimate and the initial pose estimate, generating an initial rendering of an aircraft model of the aircraft;
based on at least the video frame and the initial rendering of the aircraft model, determining refinement parameters for the initial position estimate and the initial pose estimate;
based on at least the initial position estimate, the initial pose estimate, and the refinement parameters, determining a refined position estimate and a refined pose estimate for the aircraft;
based on at least the refined position estimate and the refined pose estimate for the aircraft, determining a position and pose of a fuel receptacle on the aircraft; and
based on at least the position and pose of the fuel receptacle, controlling an aerial refueling boom to engage the fuel receptacle.
2. The method of claim 1 , wherein the video frame is provided by a single camera.
3. The method of claim 1 , wherein the refinement parameters comprise a translation refinement and a rotational refinement.
4. The method of claim 1 , further comprising:
determining the initial position estimate and the initial pose estimate based on at least the video frame showing the aircraft or a prior video frame showing the aircraft.
5. The method of claim 1 , further comprising:
determining the initial position estimate using a prior refined position estimate; and
determining the initial pose estimate using a prior refined pose estimate.
6. The method of claim 1 , further comprising:
determining a position and pose of a boom tip of the aerial refueling boom, wherein controlling the aerial refueling boom to engage the fuel receptacle comprises controlling the aerial refueling boom to engage the fuel receptacle based on at least the position and pose of the fuel receptacle and the position and pose of the boom tip.
7. The method of claim 1 , further comprising:
generating an overlay image comprising:
a model projection, based on at least the aircraft model, the refined position estimate, and the refined pose estimate; and
the video frame showing the aircraft or a subsequent video frame showing the aircraft; and
displaying the overlay image.
8. A system for aerial refueling, the system comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving a video frame showing an aircraft to be refueled;
receiving an initial position estimate and an initial pose estimate for the aircraft;
based on at least the initial position estimate and the initial pose estimate, generating an initial rendering of an aircraft model of the aircraft;
based on at least the video frame and the initial rendering of the aircraft model, determining refinement parameters for the initial position estimate and the initial pose estimate;
based on at least the initial position estimate, the initial pose estimate, and the refinement parameters, determining a refined position estimate and a refined pose estimate for the aircraft;
based on at least the refined position estimate and the refined pose estimate for the aircraft, determining a position and pose of a fuel receptacle on the aircraft; and
based on at least the position and pose of the fuel receptacle, controlling an aerial refueling boom to engage the fuel receptacle.
9. The system of claim 8 , wherein the video frame is provided by a single camera.
10. The system of claim 8 , wherein the refinement parameters comprise a translation refinement and a rotational refinement.
11. The system of claim 8 , wherein the operations further comprise:
determining the initial position estimate and the initial pose estimate based on at least the video frame showing the aircraft or a prior video frame showing the aircraft.
12. The system of claim 8 , wherein the operations further comprise:
determining the initial position estimate using a prior refined position estimate; and
determining the initial pose estimate using a prior refined pose estimate.
13. The system of claim 8 , wherein the operations further comprise:
determining a position and pose of a boom tip of the aerial refueling boom, wherein controlling the aerial refueling boom to engage the fuel receptacle comprises controlling the aerial refueling boom to engage the fuel receptacle based on at least the position and pose of the fuel receptacle and the position and pose of the boom tip.
14. The system of claim 8 , further comprising:
a boom control that controls the aerial refueling boom to engage the fuel receptacle.
15. A computer program product, comprising a non-transitory computer usable medium having a computer readable program code embodied therein, the computer readable program code adapted to be executed to implement a method of aerial refueling, the method comprising:
receiving a video frame showing an aircraft to be refueled;
receiving an initial position estimate and an initial pose estimate for the aircraft;
based on at least the initial position estimate and the initial pose estimate, generating an initial rendering of an aircraft model of the aircraft;
based on at least the video frame and the initial rendering of the aircraft model, determining refinement parameters for the initial position estimate and the initial pose estimate;
based on at least the initial position estimate, the initial pose estimate, and the refinement parameters, determining a refined position estimate and a refined pose estimate for the aircraft;
based on at least the refined position estimate and the refined pose estimate for the aircraft, determining a position and pose of a fuel receptacle on the aircraft; and
based on at least the position and pose of the fuel receptacle, controlling an aerial refueling boom to engage the fuel receptacle.
16. The computer program product of claim 15 , wherein the video frame is provided by a single camera.
17. The computer program product of claim 15 , wherein the refinement parameters comprise a translation refinement and a rotational refinement.
18. The computer program product of claim 15 , wherein the method further comprises:
determining the initial position estimate and the initial pose estimate based on at least the video frame showing the aircraft or a prior video frame showing the aircraft.
19. The computer program product of claim 15 , wherein the method further comprises:
determining the initial position estimate using a prior refined position estimate; and
determining the initial pose estimate using a prior refined pose estimate.
20. The computer program product of claim 15 , wherein the method further comprises:
determining a position and pose of a boom tip of the aerial refueling boom, wherein controlling the aerial refueling boom to engage the fuel receptacle comprises controlling the aerial refueling boom to engage the fuel receptacle based on at least the position and pose of the fuel receptacle and the position and pose of the boom tip.