Magic wand interface and other user interaction paradigms for a flying digital assistant
Methods and systems are described for new paradigms for user interaction with an unmanned aerial vehicle (referred to as a flying digital assistant or FDA) using a portable multifunction device (PMD) such as smart phone. In some embodiments, a magic wand user interaction paradigm is described for intuitive control of an FDA using a PMD. In other embodiments, methods for scripting a shot are described.
1. A method for controlling an unmanned aerial vehicle (UAV), the method comprising:
tracking, by a processing system, a pose of a mobile device in the physical environment;
identifying, by the processing system, a preset gesture based on a change in the pose of the mobile device; and
generating, by the processing system, control commands configured to cause the UAV to autonomously maneuver in a preset flight pattern corresponding to the preset gesture.
2. The method of claim 1 , further comprising:
detecting, by the processing system, the change in the pose of the mobile device in the physical environment,
wherein the change in the pose of the mobile device comprises a change in position or orientation of the mobile device in the physical environment.
3. The method of claim 2 , wherein the change in position or orientation of the mobile device is based on a motion of a user of the mobile device or a part of the user.
4. The method of claim 3 , wherein the motion of the user of the mobile device or the part of the user comprises a motion in a particular direction relative to the mobile device to cause the UAV to move in the particular direction relative to the UAV.
5. The method of claim 1 , wherein the preset flight pattern comprises one or more preset aerial maneuvers.
6. The method of claim 1 , further comprising:
identifying, by the processing system, the change in the pose of the mobile device based on the tracking; and
recognizing, by the processing system, the preset flight pattern based on the change in the pose of the mobile device.
7. The method of claim 1 , further comprising:
transmitting the control commands to the UAV.
8. The method of claim 1 , further comprising:
receiving an input indicating an intent to control the UAV; and
responsive to receiving the input indicating the intent to control the UAV, calibrating the mobile device.
9. The method of claim 8 , wherein calibrating the mobile device comprises estimating a relative position and/or orientation of the mobile device.
10. The method of claim 9 , wherein a process of visual inertial odometry is used to estimate the relative position and/or orientation of the mobile device.
11. A system for controlling an unmanned aerial vehicle (UAV), the system comprising:
a display;
one or more processors; and
one or more memory units having instructions stored thereon which, when executed by the one or more processors, cause the system to:
cause the display to present a view of a physical environment based at least in part on data gathered by sensors onboard the UAV in flight within the physical environment;
identify a preset gesture based on a change in a pose of a mobile device in the physical environment;
generate control commands configured to cause the UAV to autonomously maneuver in a preset flight pattern corresponding to the preset gesture; and
transmit the control commands to the UAV.
12. The system of claim 11 , wherein the instructions, when executed by the one or more processors, further cause the system to:
dynamically cause the display to present an updated view of the physical environment while autonomously maneuvering in the preset flight pattern corresponding to the preset gesture.
13. The system of claim 11 , wherein the instructions, when executed by the one or more processors, further cause the system to:
track the pose of the mobile device in the physical environment;
detect the change in the pose of the mobile device in the physical environment,
wherein the change in the pose of the mobile device comprises a change in position or orientation of the mobile device in the physical environment, and
wherein the change in position or orientation of the mobile device is based on a motion of a user of the mobile device or a part of the user.
14. The system of claim 13 , wherein the motion of the user of the mobile device or the part of the user comprises a motion in a particular direction relative to the mobile device to cause the UAV to move in the particular direction relative to the UAV.
15. The system of claim 11 , wherein the instructions, when executed by the one or more processors, further cause the system to:
identify the change in the pose of the mobile device based on the tracking; and
recognize the preset flight pattern based on the change in the pose of the mobile device.
16. The system of claim 15 , wherein the preset flight pattern comprises one or more preset aerial maneuvers.
17. An unmanned aerial vehicle (UAV) configured for autonomous flight through a physical environment, the UAV comprising:
a first image capture device configured to capture images of the physical environment;
a propulsion system configured to maneuver the UAV through the physical environment;
a processing system; and
an autonomous navigation system configured to:
detect a preset gesture based on a pose of a mobile device; and
cause the UAV to autonomously maneuver in accordance with the identified gesture.
18. The UAV of claim 17 , wherein to detect the preset gesture, the autonomous navigation system is further configured to:
determine a change in the pose of the mobile device in the physical environment,
wherein the change in the pose of the mobile device comprises a change in position or orientation of the mobile device in the physical environment.
19. The UAV of claim 17 , wherein the autonomous navigation system is further configured to:
track the pose of a mobile device in the physical environment.
20. The UAV of claim 18 , wherein the autonomous navigation system is configured to:
receive and process images from the first image capture device to track the pose of the mobile device.