Virtual laser pointer as a point of contact indicator for machine learning assisted aerial refueling or other targeting

1. A system comprising:

a processor; and

a non-transitory computer-readable medium having stored therein instructions that are executable to cause the system to perform functions comprising:

acquiring an image of a first aerial vehicle, wherein the image is acquired by a second aerial vehicle and depicts an object of the second aerial vehicle prior to contact between the object of the second aerial vehicle and a surface of the first aerial vehicle,

providing the image as input to a data-driven analyzer that is trained in a supervised setting with example images for determining a predicted point of contact between the object of the second aerial vehicle and the surface of the first aerial vehicle, wherein the data-driven analyzer is configured to estimate an actual point of contact between the object of the second aerial vehicle and the surface of the first aerial vehicle based on features extracted from the image;

determining, based on an output of the data-driven analyzer corresponding to the input, an estimated point of contact between the object of the second aerial vehicle and the surface of the first aerial vehicle; and

providing the estimated point of contact to a display system.

2. The system of claim 1 , wherein the data-driven analyzer comprises a parameterized model that is trained in a supervised setting by introducing a set of example images including the object of the second aerial vehicle and the surface of the first aerial vehicle that provides an output of a predicted point of contact between the object of the second aerial vehicle and the surface of the first aerial vehicle, the parameterized model being configured to estimate an actual point of contact between the object of the second aerial vehicle and the surface of the first aerial vehicle based on features extracted from the image.

3. The system of claim 1 , wherein the data-driven analyzer comprises a neural network that is trained in a supervised setting by introducing a set of example images including the object of the second aerial vehicle and the surface of the first aerial vehicle that provides an output of a predicted point of contact between the object of the second aerial vehicle and the surface of the first aerial vehicle, the neural network being configured to estimate an actual point of contact between the object of the second aerial vehicle and the surface of the first aerial vehicle based on features extracted from the image.

4. The system of claim 1 , wherein:

the first aerial vehicle is a receiver aircraft,

the second aerial vehicle is a supply aircraft, and

the object of the second aerial vehicle is a refueling boom.

5. The system of claim 1 , further comprising a camera configured to acquire the image of the second aerial vehicle.

6. The system of claim 1 , further comprising the display system, wherein the display system is configured to display a video stream of the first aerial vehicle and to overlay an indication of the estimated point of contact on the surface of the first aerial vehicle within the video stream.

7. The system of claim 6 , wherein:

the output of the data-driven analyzer corresponding to the input comprises a confidence of the estimated point of contact, and

the indication of the estimated point of contact comprises an indication of the confidence.

8. The system of claim 7 , wherein the indication of the confidence comprises a size of a shape surrounding the indication of the estimated point of contact.

9. The system of claim 7 , wherein the indication of the confidence comprises a color of the indication of the estimated point of contact.

10. The system of claim 1 :

wherein the functions further comprise:

acquiring a sequence of images that depict the first aerial vehicle and the object of the second aerial vehicle prior to contact between the object of the second aerial vehicle and the surface of the first aerial vehicle; and

providing the sequence of images as the input to the data-driven analyzer, wherein the image is part of the sequence of images, and

wherein the data-driven analyzer is configured to estimate an actual point of contact based on features extracted from the sequence of images.

11. The system of claim 1 :

wherein the functions further comprise:

acquiring an orientation and position of the first aerial vehicle relative to the second aerial vehicle; and

providing the orientation and position as part of the input to the data-driven analyzer, and

wherein the data-driven analyzer is configured to estimate an actual point of contact based on the orientation and position.

12. The system of claim 11 , wherein acquiring the orientation and position comprises deriving the orientation and position from the image.

13. The system of claim 1 :

wherein the data-driven analyzer is configured to estimate a separation distance between the object of the second aerial vehicle and the surface of the first aerial vehicle, and

wherein the functions further comprise:

determining, based on the output of the data-driven analyzer, an estimated separation distance between the object of the second aerial vehicle and the surface of the first aerial vehicle; and

providing the estimated separation distance to the display system.

14. A method comprising:

acquiring an image of a first aerial vehicle, wherein the image is acquired by a second aerial vehicle and depicts an object of the second aerial vehicle prior to contact between the object of the second aerial vehicle and a surface of the first aerial vehicle;

providing the image as input to a data-driven analyzer that is trained in a supervised setting with example images for determining a predicted point of contact between the object of the second aerial vehicle and the surface of the first aerial vehicle, wherein the data-driven analyzer is configured to estimate an actual point of contact between the object of the second aerial vehicle and the surface of the first aerial vehicle based on features extracted from the image;

determining, based on an output of the data-driven analyzer corresponding to the input, an estimated point of contact between the object of the second aerial vehicle and the surface of the first aerial vehicle; and

overlaying an indication of the estimated point of contact on the surface of the first aerial vehicle within a video stream of the first aerial vehicle.

15. The method of claim 14 , wherein:

the first aerial vehicle is a receiver aircraft,

the second aerial vehicle is a supply aircraft, and

the object of the second aerial vehicle is a refueling boom.

16. The method of claim 14 , wherein:

the output of the data-driven analyzer corresponding to the input comprises a confidence of the estimated point of contact, and

the indication of the estimated point of contact comprises an indication of the confidence.

17. The method of claim 14 , further comprising:

acquiring a sequence of images that depict the first aerial vehicle and the object of the second aerial vehicle prior to contact between the object of the second aerial vehicle and the surface of the first aerial vehicle; and

providing the sequence of images as the input to the data-driven analyzer,

wherein the image is part of the sequence of images, and

wherein the data-driven analyzer is configured to estimate an actual point of contact based on features extracted from the sequence of images.

18. The method of claim 14 :

wherein the data-driven analyzer is configured to estimate a separation distance between the object of the second aerial vehicle and the surface of the first aerial vehicle, and

wherein the method further comprises:

determining, based on the output of the data-driven analyzer, an estimated separation distance between the object of the second aerial vehicle and the surface of the first aerial vehicle; and

displaying the estimated separation distance within the video stream.

19. A non-transitory computer-readable medium having stored therein instructions that are executable to cause a system to perform functions comprising:

acquiring an image of a first aerial vehicle, wherein the image is acquired by a second aerial vehicle and depicts an object of the second aerial vehicle prior to contact between the object of the second aerial vehicle and a surface of the first aerial vehicle;

providing the image as input to a data-driven analyzer that is trained in a supervised setting with example images for determining a predicted point of contact between the object of the second aerial vehicle and the surface of the first aerial vehicle, wherein the data-driven analyzer is configured to estimate an actual point of contact between the object of the second aerial vehicle and the surface of the first aerial vehicle based on features extracted from the image;

determining, based on an output of the data-driven analyzer corresponding to the input, an estimated point of contact between the object of the second aerial vehicle and the surface of the first aerial vehicle; and

overlaying an indication of the estimated point of contact on the surface of the first aerial vehicle within a video stream of the first aerial vehicle.

20. The non-transitory computer-readable medium of claim 19 , wherein:

the first aerial vehicle is a receiver aircraft,

the second aerial vehicle is a supply aircraft, and

the object of the second aerial vehicle is a refueling boom.