Aircraft flight user interface
A graphical user interface (GUI) for controlling the flight of an aircraft such as an unmanned aerial vehicle (UAV) is described. In example embodiments, the GUI includes a view of a physical environment from the perspective of the aircraft as well as various interactive elements through which a user can interact. The interactive elements presented in the GUI and the way in which user interaction with such elements is interpreted can depend on user selection from multiple available control modes. In some embodiments, user interaction with the GUI is interpreted based on a selected control mode and translated into behavioral objectives that are processed, along with other behavioral objectives, by a motion planner to maneuver the aircraft.
1 . A method for controlling an aircraft, the method comprising:
receiving, by a computer system, a selection of a control mode of a plurality of available control modes;
presenting, by the computer system, a graphical user interface (GUI) at a user computing device,
wherein the graphical user interface includes a display of a view of the physical environment from a perspective of the aircraft and an arrangement of one or more interactive elements, the view generated based on sensor data from a sensor device onboard the aircraft, and the arrangement of the one or more interactive elements determined based on the selected control mode;
detecting, by the computer system, a user interaction with the GUI;
responsive to detecting the user interaction with the GUI, identifying, by the computer system, the selected control mode of the plurality of available control modes,
wherein the manner in which the aircraft responds to the user interaction is dependent on the selected control mode;
interpreting, by the computer system, the user interaction as indicative of a particular input, command or intention associated with the selected control mode,
wherein the user interaction is interpreted differently based on each of the plurality of available control modes;
translating, by the computer system, the interpreted user interaction into a behavioral objective that includes one or more parameter values including a target, a dead-zone region, and a weighting factor;
generating, by a motion planner of a navigation system of the aircraft, a planned trajectory through the three-dimensional space based on the behavioral objective;
generating, by the computer system, control commands for causing the aircraft to fly along the planned trajectory;
displaying, by the computer system, in the GUI, a graphical representation of the planned trajectory overlaid on the view of the physical environment,
wherein the arrangement of the interactive elements dynamically changes based on the selected control mode, and user interaction with the interactive elements is interpreted differently across different control modes; and
continually updating the planned trajectory based on perception inputs including object-tracking data for a tracked subject, such that movement of the aircraft dynamically follows movement of the tracked subject.
2 . The method of claim 1 , wherein based on the selected control mode, the user interaction is interpreted as a user selection of a physical object in the physical environment that is represented in the displayed view of the physical environment in the GUI.
3 . The method of claim 1 , further comprising:
detecting, by the computer system, an obstacle in the physical environment;
determining, by the computer system, based on the behavioral objective that the planned trajectory will cause the aircraft to collide with the obstacle;
adjusting, by the computer system, the planned trajectory so as to avoid collision with the obstacle; and
displaying, by the computer system, in the GUI, an indication of the detected obstacle.
4 . The method of claim 1 , further comprising:
changing, by the computer system, an arrangement of one or more interactive elements in the GUI based on the selected control mode.
5 . The method of claim 1 , wherein the user interaction is detected as a touch gesture via a touch screen display of the user computing device.
6 . The method of claim 1 , wherein the user computing device is any of a smart phone, a tablet device, an augmented reality device, or a virtual reality device.
7 . The method of claim 1 , wherein the aircraft is an unmanned aerial vehicle (UAV).
8 . The method of claim 1 , wherein the sensor is an image capture device and wherein the view of the physical environment includes a rendering of a three-dimensional (3D) model of the physical environment, the 3D model generated based on images captured by the image capture device.
9 . A system for controlling an unmanned aerial vehicle (UAV), the system comprising:
an interactive display device;
an interactive menu that enables a user to select from a plurality of different control modes;
a processor; and
a memory having instructions stored thereon, which when executed by the processor, cause the system to:
display, using the interactive display device, a graphical user interface comprising:
a view of the physical environment based on sensor data from a sensor device onboard the UAV; and
a plurality of interactive elements that enable the user to interact with the GUI to control the UAV;
wherein an arrangement of the interactive elements dynamically changes based on a selected control mode of the plurality of different control modes;
detect a user interaction with the GUI;
responsive to the user interaction, identify the selected control mode, wherein the manner in which the aircraft responds to the user interaction is dependent on the selected control mode;
interpret the detected user interaction as indicative of a particular input, command or intention associated with the selected control mode,
wherein the arrangement of interactive elements dynamically changes based on the selected control mode and the interpretation of user interactions differs across control modes;
translate the interpreted user interaction into a behavioral objective comprising parameter values including a target, a dead-zone region, and a weighting factor; and
input the behavioral objective into a motion planner configured to generate a planned trajectory through three-dimensional space and to continually update the planned trajectory based on perception inputs including object-tracking data for a tracked subject;
wherein an interactive element of the plurality of interactive elements dynamically changes from active to inactive in response to a determination that a maneuver by the UAV will cause a collision with an obstacle.
10 . The system of claim 9 , wherein the interactive elements include one or more of a virtual button, a virtual joystick, a slider bar, or an interactive graphical overlay.
11 . The system of claim 10 , wherein the interactive graphical overlay is displayed over a portion of the view of the physical environment at a location corresponding to a representation of a detected physical object in the physical environment that is captured in the view.
12 . The system of claim 9 , wherein the view of the physical environment includes any of:
a live video feed from an image capture device onboard the UAV; or
a rendering of a three-dimensional (3D) model of the physical environment from a perspective corresponding to a position of the UAV in the physical environment, the 3D model generated based on the sensor data.
13 . The system of claim 9 , wherein the plurality of different user-selectable control modes includes a plurality of user-selectable modes of operation and a plurality of user-selectable cinematic modes.
14 . The system of claim 13 , wherein the selected control mode represents a combination of a selected mode of operation and a selected cinematic mode.
15 . The system of claim 9 , wherein the interactive display device includes a touch-sensitive display.
16 . The system of claim 9 , wherein an interactive element of the plurality of interactive elements dynamically changes from active to inactive in response to a determination that a maneuver by the UAV based on user interaction with the interactive element will cause a collision with an obstacle.
17 . The system of claim 9 , wherein the GUI further comprises:
a graphical representation of the planned trajectory overlaid on the view of the physical environment.
18 . The system of claim 9 , wherein an interactive element of the plurality of interactive elements is overlaid on the view of the physical environment.
19 . An aircraft flight control system comprising:
a navigation system for:
continually generating and updating a planned trajectory through three-dimensional space based on perception inputs, one or more behavioral objectives comprising parameter values including at least one of a target, a dead-zone region, and a weighting factor, and a tracked motion of a physical object; and
controlling one or more control actuators onboard the aircraft to cause the aircraft to autonomously fly through a physical environment along the planned trajectory; and
a graphical user interface (GUI) coupled to the navigation system to enable a user to define behavioral objectives for processing by the navigation system, the GUI comprising:
a view of the physical environment based on the perception inputs;
a representation of the planned trajectory overlaid on the view of the physical environment; and
an interactive element that dynamically changes based on a selected control mode of a plurality of different user-selectable control modes;
wherein user interaction with the interactive element is interpreted differently based on the selected control mode and is translated into the behavioral objective for processing by a motion planner of the navigation system to control the aircraft.
20 . The aircraft flight control system of claim 19 , wherein the GUI further comprises:
an interactive menu that enables the user to select from the plurality of different control modes.
21 . The aircraft flight control system of claim 20 , wherein the interactive element includes any of a virtual button, a virtual joystick, a slider bar, or an interactive graphical overlay.
22 . The aircraft flight control system of claim 19 , wherein the view of the physical environment includes any of:
a live video feed from an image capture device onboard the aircraft; or
a rendering of a three-dimensional (3D) model of the physical environment from a perspective corresponding to a position of the aircraft in the physical environment, the 3D model generated based on sensor data from sensors onboard the aircraft.
23 . The aircraft flight control system of claim 19 , wherein the plurality of different user-selectable control modes includes a plurality of different user-selectable modes of operation and a plurality of different user-selectable cinematic modes.
24 . The aircraft flight control system of claim 23 , wherein the selected control mode represents a combination of a selected mode of operation from the plurality of different user-selectable modes of operation and a selected cinematic mode from the plurality of different user-selectable cinematic modes.
25 . The aircraft flight control system of claim 19 , wherein the GUI is presented to the user via an interactive display device.
26 . The aircraft flight control system of claim 25 , wherein the interactive display device is integrated into a mobile device in wireless communication with the aircraft.
27 . The aircraft flight control system of claim 19 , wherein the GUI further comprises:
a graphical representation of the planned trajectory generated by the navigation system overlaid on the view of the physical environment.
28 . The aircraft flight control system of claim 19 , wherein the interactive element is overlaid on the view of the physical environment.
29 . The aircraft flight control system of claim 19 , wherein the planned trajectory is generated by the navigation system to avoid obstacles regardless of any user interaction with the GUI.
30 . The aircraft flight control system of claim 19 , wherein the interactive element includes a virtual joystick, and wherein user interaction with the virtual joystick causes the navigation system to update the planned trajectory to cause the aircraft to fly in a direction corresponding to the user interaction with the virtual joystick at a constant altitude.
31 . The aircraft flight control system of claim 19 , wherein the interactive element includes a subject selection element overlaid on a portion of the view corresponding with a representation of a detected physical object in the physical environment, and wherein user interaction with the subject selection element causes the navigation system to track the detected physical object and update the planned trajectory to cause the aircraft to follow the detected object.
32 . The aircraft flight control system of claim 19 , wherein the displayed view of the physical environment is interactive and wherein user interaction with the displayed view causes the navigation system to update the planned trajectory and/or control an orientation of a gimbaled image capture device onboard the aircraft.
33 . The aircraft flight control system of claim 32 , wherein user interaction with the displayed view is interpreted as a selection of a particular point in the physical environment that is represented in the displayed view, and wherein the planned trajectory is updated to cause the aircraft to maneuver towards a position of the particular point in the physical environment, and wherein orientation of the gimbaled image capture device is adjusted to keep the particular point in the physical environment within the displayed view as the aircraft maneuvers along the planned trajectory.
34 . The aircraft flight control system of claim 19 , wherein the GUI further comprises:
an indication of an obstacle that is displayed in response to the navigation system detecting the obstacle based on the perception inputs.
35 . The aircraft flight control system of claim 19 , wherein the GUI further comprises:
a graphical representation of the planned trajectory overlaid on the view of the physical environment.
36 . The aircraft flight control system of claim 19 , wherein the GUI further comprises:
a computer-generated 3D occupancy map overlaid on the view of the physical environment.
37 . The method of claim 3 , further comprising:
displaying, by the computer system, in the GUI, an indication of a divergence from the planned trajectory to avoid collision with the obstacle.
38 . The method of claim 3 , further comprising:
displaying, by the computer system, in the GUI, an indication that the behavioral objective based on the user interaction cannot be satisfied due to the detected obstacle.