User interaction paradigms for a flying digital assistant
Methods and systems are described for new paradigms for user interaction with an unmanned aerial vehicle (referred to as a flying digital assistant or FDA) using a portable multifunction device (PMD) such as smart phone. In some embodiments, a user may control image capture from an FDA by adjusting the position and orientation of a PMD. In other embodiments, a user may input a touch gesture via a touch display of a PMD that corresponds with a flight path to be autonomously flown by the FDA.
1. A method for capturing aerial video of a physical environment with synchronized distributed audio capture, the method comprising:
receiving, by a wireless network interface of a first mobile device, video of the physical environment captured by a camera of an unmanned aerial vehicle (UAV) in autonomous flight over the physical environment;
capturing, by a first microphone of the first mobile device, first audio of the physical environment; and
synchronizing, by a processing unit of the first mobile device:
the video of the physical environment captured by the camera of the UAV; and
the first audio of the physical environment captured by the first microphone of the first mobile device.
2. The method of claim 1 , further comprising:
receiving, by the wireless network interface of the first mobile device, second audio of the physical environment captured by a second microphone; and
synchronizing, by the processing unit of the first mobile device:
the video of the physical environment captured by the camera of the UAV;
the first audio of the physical environment captured by the first microphone of the first mobile device; and
the second audio of the physical environment captured by the second microphone.
3. The method of claim 2 , wherein the second microphone is associated with any of the UAV, a second UAV that is difference than the UAV, or a second mobile device that is different than the first mobile device.
4. The method of claim 1 , wherein the first mobile device is within a threshold proximity to the UAV.
5. The method of claim 1 , further comprising:
determining that the first mobile device is within a threshold proximity to the UAV before synchronizing the video and the first audio.
6. The method of claim 1 , further comprising:
receiving, by the wireless network interface of the first mobile device, second audio of the physical environment captured by a plurality of microphones of a distributed network of mobile devices; and
synchronizing, by the processing unit of the first mobile device:
the video of the physical environment captured by the camera of the UAV;
the first audio of the physical environment captured by the first microphone of the first mobile device; and
the second audio of the physical environment captured by the plurality of microphones of the distributed network of mobile devices.
7. The method of claim 1 , wherein the first audio of the physical environment is captured, by the first microphone of the first mobile device, in real time or near real time as the video of the physical environment is captured by the camera of the UAV.
8. The method of claim 1 , wherein the video and the first audio are synchronized in real time or near real time as the video and first audio are captured.
9. The method of claim 1 , wherein the video and the first audio are synchronized while the UAV is still in flight through the physical environment.
10. The method of claim 1 , wherein the video and the first audio are synchronized to be temporally consistent with each other.
11. The method of claim 1 , wherein the first mobile device is any of a smart phone, a tablet computer, a dedicated remote-control device associated with the UAV, or a second UAV that is different than the UAV.
12. The method of claim 1 , further comprising:
generating a video file that includes the video and the first audio after synchronizing the video and the first audio.
13. A method for capturing aerial video of a physical environment with synchronized distributed audio capture, the method comprising:
capturing, by a camera coupled to an unmanned aerial vehicle (UAV), video of the physical environment;
receiving, via a wireless network interface of the UAV, first audio of the physical environment captured by first microphone coupled to a first mobile device; and
synchronizing, by a processing unit of the UAV:
the video of the physical environment captured by the camera of the UAV; and
the first audio of the physical environment captured by the first microphone of the first mobile device.
14. The method of claim 13 , further comprising:
capturing, by a second microphone coupled to the UAV, second audio of the physical environment; and
synchronizing, by the processing unit of the UAV:
the video of the physical environment captured by the camera of the UAV;
the first audio of the physical environment captured by first microphone of the first mobile device; and
the second audio of the physical environment captured by the second microphone coupled to the UAV.
15. The method of claim 13 , further comprising:
capturing, by a second microphone of a second mobile device, second audio of the physical environment, wherein the second mobile device is different than the first mobile device; and
synchronizing, by the processing unit of the UAV:
the video of the physical environment captured by the camera of the UAV;
the first audio of the physical environment captured by first microphone of the first mobile device; and
the second audio of the physical environment captured by the second microphone coupled of the second mobile device.
16. The method of claim 15 , wherein the first mobile device and second mobile device are each one of a plurality of mobile devices in a distributed network of mobile devices.
17. The method of claim 13 , further comprising:
determining that the first device is within a threshold proximity to the UAV before synchronizing the video with the first audio.
18. The method of claim 13 , wherein the first audio of the physical environment is captured, by the first microphone of the first mobile device, in real time or near real time as the video of the physical environment is captured by the camera of the UAV.
19. The method of claim 13 , wherein the video and the first audio are synchronized in real time or near real time as the video and first audio are captured.
20. The method of claim 13 , wherein the video and the first audio are synchronized while the UAV is still in flight through the physical environment.
21. The method of claim 13 , wherein the video and the first audio are synchronized to be temporally consistent with each other.
22. The method of claim 13 , further comprising:
in real time or near real time as the video and first audio are synchronized:
generating a video file based on the synchronization of the video and the first audio; and
streaming, via the wireless network interface of the UAV, the video file for presentation at a display device.
23. The method of claim 22 , wherein the display device is associated with the first mobile device.
24. An unmanned aerial vehicle (UAV) configured to capture aerial video of a physical environment with synchronized distributed audio capture, the UAV comprising:
an image capture device configured to capture video of the physical environment as the UAV is in flight through the physical environment;
a wireless communication interface configured to receive first audio data from one or more mobile devices located in the physical environment as the UAV is in flight through the physical environment, the first audio data indicative of audio of the physical environment captured by the one or more mobile devices; and
a processing unit configured to synchronize the video with the first audio data in real time or near real time as the video is captured and as the first audio data is received.
25. The UAV of claim 24 , further comprising:
a microphone configured to capture second audio of the physical environment as the UAV is in flight through the physical environment;
wherein the processing unit is further configured to synchronize the video with the first audio and the second audio in real time or near real time as the video is captured, as the first audio data is received, and as the second audio data is captured.
26. The UAV of claim 24 ,
wherein the processing unit is further configured to generate an audio-synchronized video based on the synchronization of the video with the first audio; and
wherein the wireless communication interface is configured to stream, as the UAV is in flight through the physical environment, the audio-synchronized video for presentation at a remote display device.
27. The UAV of claim 24 , the remote display device being associated with one of the one or more mobile devices.
28. The UAV of claim 24 ,
wherein the processing unit is further configured to:
determine a position of the UAV and the one or more mobile devices;
determine, based on the positions of the UAV and the one or more mobile devices, that the one or more mobile devices are within a threshold proximity to the UAV; and
wherein the processing unit synchronizes the video with the first audio in response to determining that the one or more mobile devices are within the threshold proximity to the UAV.
29. The UAV of claim 24 , wherein the processing unit is further configured to cause the UAV to autonomously maneuver through the physical environment based on the video captured by the image capture device.
30. The UAV of claim 24 , wherein the processing unit is configured to synchronize the video and the first audio to be temporally consistent with each other.