System and method for calculation and display of formation flight information on augmented reality display device
View Patent ↗Disclosed are systems and associated methods for calculating and displaying formation flight information, to include aircraft spacing, predicted trajectory data, and lost visual contact alerts, on an augmented reality display designed to interface with aviation helmets. The augmented reality display device collects imagery of a leading aircraft from the vantage point of a pilot in a trailing aircraft. The collected imagery is processed through object detection, tracking, and orientation estimation techniques that return the leading aircraft's distance, direction, and orientation. The estimated leading aircraft data informs a generated virtual overlay. The generated virtual overlay is transmitted to the augmented reality display for viewing by the pilot in the trailing aircraft. The display of relevant formation flight information using augmented reality tools may result in improved formation flight spacing, emergency procedure response, and collision avoidance.
1. A system for calculating and displaying formation flight information through an augmented reality display device on a trailing aircraft in a flight formation, the system comprising:
a helmet wearable by a pilot of an aircraft including one or more outward-facing cameras collecting imagery from the vantage point of a pilot in the trailing aircraft;
one or more augmented reality displays;
a processor configured to estimate the distance, direction, and orientation of a leading aircraft based on the imagery of one or more outward-facing cameras; and
a virtual overlay program for displaying information relating to leading aircraft distance, direction, and orientation on the augmented reality display.
2. The system of claim 1 , wherein information relating to leading aircraft distance, direction, and orientation is presented as text or symbology, and optionally comprise aircraft spacing, the predicted trajectory of the leading aircraft, and/or lost visual contact alerts.
3. The system of claim 1 , wherein the pilot is provided an option to differentiate between being a pilot on the controls and a pilot managing the cockpit, in the case of a two-pilot aircraft for which to display from the vantage point thereof using the virtual overlay program.
4. The system of claim 1 , wherein the estimation of leading aircraft distance and direction is accomplished by the processor using object detection and tracking techniques.
5. The system of claim 1 , wherein the estimation of the leading aircraft pitch, bank, and roll angles by the processor is performed using orientation estimation and object tracking techniques.
6. The system of claim 1 , further comprising one or more pressure sensors; and
wherein the processor is further configured to determine leading and trailing aircraft altitude as a function of the outputs of one or more pressure sensors and leading aircraft distance and direction.
7. The system of claim 1 , further comprising one or more global positioning system (GPS) receivers; and wherein the processor is further configured to determine leading and trailing aircraft location as a function of the outputs of one or more GPS receivers and leading aircraft distance and direction.
8. The system of claim 1 , further comprising a plurality of gyroscopes and one or more compasses; and wherein the processor is further configured to detect and track the leading aircraft from the vantage point of the pilot in the trailing aircraft, as a function of the outputs of the plurality of gyroscopes, one or more compasses, and leading aircraft distance and direction.
9. A method for determining and displaying formation flight lost visual contact alerts using an augmented reality display device, the method comprising:
collecting imagery from the vantage point of a pilot in a trailing aircraft using one or more outward-facing cameras;
determining the presence of a leading aircraft from the vantage point of the pilot in the trailing aircraft using a processor running an object detection neural network; and
issuing a lost visual contact alert to an augmented reality display if the processor determines that the leading aircraft is not visible from the vantage point of the pilot in the trailing aircraft.
10. The method of claim 9 , wherein the relative positioning of the leading aircraft from the vantage point of the pilot in the trailing aircraft is further determined and tracked using mapping supported by a plurality of gyroscopes and/or one or more compasses.
11. The method of claim 9 , wherein the relative positioning of the leading aircraft from the vantage point of the pilot in the trailing aircraft is further determined and tracked using object tracking techniques.
12. The method of claim 9 , wherein the issued lost visual contact alert displays information is defined by a user input form.
13. The method of claim 9 , wherein the issued lost visual contact alert contains a directionality vector denoting the relative position of the leading aircraft outside of the immediate view of the pilot in the trailing aircraft.
14. A method for calculating and displaying formation flight information using an augmented reality display device, the method comprising:
collecting imagery of a leading aircraft from the vantage point of a pilot in a trailing aircraft using one or more outward-facing cameras mounted in or on a helmet wearable by a pilot of an aircraft;
estimating leading aircraft distance and direction using object detection and tracking techniques processing imagery of the leading aircraft from the vantage of point of the pilot in the trailing aircraft using a processor;
estimating aircraft relative position as a function of leading aircraft distance and direction using the processor; and
displaying information relating to aircraft relative position on an augmented reality display device using a virtual overlay program generated by the processor.
15. The method of claim 14 , wherein information relating to aircraft relative position is presented as text and/or symbology.
16. The method of claim 14 , wherein information relating to aircraft relative position comprises aircraft spacing and/or lost visual contact alerts.
17. The method of claim 14 , further comprising: providing an option for a pilot to differentiate between being a pilot on the controls and a pilot managing the cockpit, in the case of a two-pilot aircraft for which to display from the vantage point thereof using the virtual overlay program.
18. The method of claim 14 , wherein the object detection and tracking techniques are aircraft specific.
19. A method for calculating and displaying formation flight information using an augmented reality display device, the method comprising:
collecting imagery of a leading aircraft from the vantage point of a pilot in a trailing aircraft using one or more outward-facing cameras mounted in or on a helmet wearable by a pilot of an aircraft; and
estimating, using at least one processor, leading aircraft bank angle, pitch angle, and heading using orientation estimation techniques processing imagery of the leading aircraft from the vantage of point of the pilot in the trailing aircraft; and
estimating, using the at least one processor, a predicted trajectory of the leading aircraft as a function of leading aircraft bank angle, pitch angle, and heading; and
displaying the predicted trajectory of the leading aircraft on an augmented reality display device using a virtual overlay program generated by the at least one processor.
20. The method of claim 19 , wherein the leading aircraft and/or predicted trajectory thereof is presented as text or symbology.
21. The method of claim 19 , further comprising: providing an option for a pilot to differentiate between being a pilot on the controls and a pilot managing the cockpit, in the case of a two-pilot aircraft for which to display from the vantage point thereof using the virtual overlay program.
22. The method of claim 19 , wherein the orientation estimation techniques are aircraft specific, and wherein the method further comprises: providing a digital user input form to a pilot to select the aircraft type.