Magic wand interface and other user interaction paradigms for a flying digital assistant
Methods and systems are described for new paradigms for user interaction with an unmanned aerial vehicle (referred to as a flying digital assistant or FDA) using a portable multifunction device (PMD) such as smart phone. In some embodiments, a magic wand user interaction paradigm is described for intuitive control of an FDA using a PMD. In other embodiments, methods for scripting a shot are described.
1 . A method for controlling an unmanned aerial vehicle (UAV), the method comprising:
tracking, by a processing system onboard the UAV, a pose or motion of a subject in a physical environment using image-based feature tracking, passive wireless localization, or inertial measurements of the UAV;
identifying, by the processing system, a preset gesture based on a detected change in the pose or motion of the subject, including correlating visual-odometry-derived feature motion with UAV inertial measurements to estimate a relative trajectory between the UAV and the subject; and
generating, onboard the UAV, control commands configured to cause the UAV to autonomously maneuver in a preset flight pattern corresponding to the preset gesture, the preset flight pattern being constrained to maintain a substantially constant radial distance from the subject in a spherical coordinate system; and
autonomously maneuvering the UAV in accordance with the generated control commands.
2 . The method of claim 1 , wherein tracking the subject includes detecting the subject using computer vision-based human recognition.
3 . The method of claim 1 , wherein tracking the subject includes device-free passive localization using changes to received wireless signals as the subject moves.
4 . The method of claim 1 , wherein identifying the preset gesture includes detecting arm-sweep motions of the subject.
5 . The method of claim 1 , wherein generating control commands includes applying feed-forward smoothing to predict the subject's intended trajectory.
6 . The method of claim 1 , wherein the UAV dynamically modifies the preset flight pattern based on detection of obstacles in the environment.
7 . The method of claim 1 , wherein the UAV maintains framing of the subject using a hybrid mechanical-digital gimbal.
8 . The method of claim 1 , wherein the UAV applies a visual-inertial odometry algorithm to estimate relative motion between the UAV and the subject.
9 . The method of claim 1 , wherein identifying the preset gesture includes detecting an azimuth-change gesture or an inclination-change gesture.
10 . The method of claim 1 , wherein the UAV maintains constant radial distance by constraining motion in a cylindrical coordinate system.
11 . A system for controlling an unmanned aerial vehicle (UAV), the system comprising:
a display;
one or more processors; and
one or more memory units having instructions stored thereon which, when executed by the one or more processors, cause the system to:
present a real-time three-dimensional reconstructed model of a physical environment generated using sensor data from the UAV;
identify a preset gesture of a subject based on a change in the subject's pose or motion detected from the reconstructed model;
interpret the preset gesture as a virtual-camera manipulation within the reconstructed model and generate control data defining a preset flight pattern corresponding to the manipulation, the preset flight pattern being dynamically modifiable; and
transmit the control data to the UAV.
12 . The system of claim 11 , wherein the reconstructed model comprises a textured mesh generated from depth and visual odometry data.
13 . The system of claim 11 , wherein the preset gesture includes a body-lean, arm-raise, or directional step detected within the model.
14 . The system of claim 11 , wherein the virtual-camera manipulation comprises orbit, pull-out, or sweep around a subject.
15 . The system of claim 11 , wherein the preset flight pattern is automatically replanned to maintain collision-free trajectories.
16 . The system of claim 11 , wherein the preset flight pattern comprises one or more preset aerial maneuvers.
17 . An unmanned aerial vehicle (UAV), comprising:
a first image capture device;
a propulsion system;
one or more processors; and
an autonomous navigation system configured to:
detect a preset gesture of a subject based on a change in the subject's pose or motion in a physical environment using image-based tracking, passive wireless localization, or depth estimation;
determine a relative position of the subject with respect to the UAV; and
autonomously maneuver the UAV in accordance with the detected gesture while maintaining a substantially constant radial distance from the subject in a spherical or cylindrical coordinate system.
18 . The UAV of claim 17 , wherein gesture detection includes using stereo depth estimation from the first image capture device.
19 . The UAV of claim 17 , wherein gesture detection uses optical-flow vectors of tracked subject features.
20 . The UAV of claim 17 , wherein maintaining constant radial distance includes adjusting UAV yaw and pitch while stabilizing captured images using a single-axis mechanical gimbal.
21 . The UAV of claim 17 , wherein the UAV autonomously recognizes a gesture to reposition and a gesture to change speed.
22 . The UAV of claim 17 , wherein the UAV generates a preset flight pattern using a learned model of prior subject gestures.