IP Library › Granted Patent US 11,649,052
Granted Patent B2
US 11,649,052 · App. 17/196,837 · Granted May 16, 2023

System and method for providing autonomous photography and videography

Inventors: Mengqiu Wang (HangZhou, CN); Lixin Liu (HangZhou, CN); Tong Zhang (HangZhou, CN); Jia Lu (HangZhou, CN)
Assignee: Hangzhou Zero Zero Technology Co., Ltd.
B64C39/024B64D47/08G01S3/7864G01S17/86G01S17/89G05D1/0011G05D1/0088G05D1/0094G06V20/17G06V40/166H04N7/185H04N23/60H04N23/62H04N23/66B64U2101/30B64U2201/10B64U2201/20G01S17/02G06V20/13G06V20/58G06V2201/07
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,649,052
App. No.
17/196,837
Granted
May 16, 2023
Kind
B2
Abstract

An aerial system, including a processing system, an optical system, an actuation system and a lift mechanism, includes an autonomous photography and/or videography system 70 , implemented, at least in part, by the processing system 22 , the optical system 26 , the actuation system 28 and the lift mechanism 32 . The autonomous photograph and/or videography system performs the steps of establishing a desired flight trajectory, detecting a target, controlling the flight of the aerial system as a function of the desired flight trajectory relative to the target using the lift mechanism and controlling the camera to capture pictures and/or video.

Claims (54)

1. An aerial system, comprising:

a body;

a lift mechanism coupled to the body;

an optical system controllably mounted to the body by an actuation system;

an inertial measurement unit coupled to the body; and

a processing system coupled to the lift mechanism, the optical system, the actuation system, and the inertial measurement unit, the processing system includes a processor configured to:

control a flight of the aerial system to maintain the aerial system to track a moving target; and

control, through the actuation system, the optical system to make relative movements in relation to the body to capture pictures and/or video of the moving target, wherein the optical system keeps the moving target within a desired position within a frame of the pictures and/or video;

wherein the processing system includes an auto-selection module, the auto-selection module being configured to automatically select picture(s) and/or video and/or truncated video clip(s) based on predetermined criteria, and the predetermined criteria includes a rule that if a center of the moving target is not within the desired position and if an exposure value of the target in the pictures and/or video, and blurriness and image-stability of the target in the pictures and/or video is not within the desired threshold, then the selected picture is considered as a bad candidate;

wherein the auto-selection module includes a lower frequency vibration filter to check a 2D trajectory of the moving target for proceeding the rule, and a vibration detector to utilize an IMU data from the inertial measurement unit to calculate a vibration of the body.

2. The aerial system, as set forth in claim 1 , wherein the processor is further configure to:

(a) following the moving target from behind at a fixed distance, indefinitely or for a finite amount of time;

(b) leading the moving target at the front while aiming at the moving target, keeping a fixed distance, indefinitely or for a finite amount of time; or

(c) orbiting around the moving target at a fixed distance with a constant or varying speed, indefinitely or for a finite amount of time.

3. The aerial system, as set forth in claim 1 , wherein the processor is further configure to:

(a) moving closer to or further away from the moving target at a specified camera aiming angle, with a constant/varying speed, for a finite amount of time;

(b) moving in a certain direction, in world coordinates or in target coordinates, while the optical system is aimed the moving target, with a constant/varying speed, for a finite amount of time;

(c) holding one or more degrees of freedom associated with the aerial system constant and controlling other degrees of freedom to track the moving target; or,

(d) following a pre-recorded trajectory relative to the moving target.

4. The aerial system, as set forth in claim 1 , further including one or more position sensors, wherein the processing system is configured to integrate positioning data from the position sensors to establish a position of the aerial system.

5. The aerial system, as set forth in claim 1 , wherein the processing system further includes an auto-detection and tracking module, the auto-detection and tracking module being configured to:

detect and locate the moving target from a video feed received from the optical system;

and, responsively establish the position and the velocity of the moving target relative to the aerial system.

6. The aerial system, as set forth in claim 1 , wherein the predetermined criteria includes over/under exposure and/or composition.

7. The aerial system, as set forth in claim 1 , wherein the processing system includes an auto-editing module, the auto-editing module being configured to edit the selected picture(s), video, and/or video clip(s) based on predetermined editing parameters.

8. A method for operating an aerial system, the aerial system including a body, a lift mechanism, an optical system movably controlled by an actuation system mounted on the body, and an inertial measurement unit coupled to the body, the method including the steps of:

selecting a set of predefined control strategies;

establishing a current position of the aerial system;

establishing a current velocity of a moving target relative to the aerial system;

controlling a flight of the aerial system to maintain the aerial system at a fixed or varying distance from the moving target as a function of the established current position and current velocity of the moving target using the lift mechanism;

controlling the optical system through the actuation system to make relative movements in relation to the body to capture pictures and/or video of the moving target, wherein the optical system keeps the moving target within a desired position within a frame of the pictures and/or video; and

automatically selecting picture(s) and/or video and/or truncated video clip(s) based on predetermined criteria, wherein the predetermined criteria includes a rule that if a center of the moving target is not within the desired position and if an exposure value of the target in the pictures and/or video, and blurriness and image-stability of the target in the pictures and/or video is not within the desired threshold, then the selected picture is considered as a bad candidate;

checking a 2D trajectory of the moving target by a lower frequency vibration filter for proceeding the rule; and

calculating a vibration of the body, through a vibration detector, by utilizing an IMU data from the inertial measurement unit.

9. A method, as set forth in claim 8 , the set of predefined control strategies including:

following the moving target from behind at a fixed distance, indefinitely or for a finite amount of time;

leading the moving target at front while aiming at the moving target, keeping a fixed distance, indefinitely or for a finite amount of time; or

orbiting around the moving target with a constant or varying speed, indefinitely or for a finite amount of time.

10. The method, as set forth in claim 8 , the set of predefined control strategies including:

moving closer to or further away from the moving target at a specified camera aiming angle, with a constant/varying speed, for a finite amount of time;

moving in a certain direction, in world coordinates or in target coordinates, while the optical system is aimed the moving target, with a constant/varying speed, for a finite amount of time;

holding one or more degrees of freedom associated with the aerial system constant and controlling other degrees of freedom to track the moving target; or,

following a pre-recorded trajectory relative to the moving target.

11. The method, as set forth in claim 8 , wherein the set of predefined control strategies are selected by the user.

12. The method, as set forth in claim 8 , wherein the set of predefined control strategies are selected by a processor.

13. The method, as set forth in claim 8 , further including

establishing a current shooting mode from a set of predefined shooting modes, wherein the optical system makes relative movement in relation to the body as a function of the established current shooting mode.

14. The method, as set forth in claim 13 , wherein set of predefined shooting modes are presented to a user and the established shooting mode is selected by the user.

15. The method, as set forth in claim 8 , including the steps of:

integrating positioning data from position sensors to establish a position of the aerial system.

16. The method, as set forth in claim 8 , including the steps of:

detecting and locating the moving target from a video feed received from the optical system; and, responsively establishing the position and the velocity of the moving target relative to the aerial system.

17. The method, as set forth in claim 8 , wherein the predetermined criteria includes over/under exposure and/or composition.

18. The method, as set forth in claim 17 , including editing the selected picture(s), video, and/or video clip(s) based on predetermined editing parameters.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 11, 2021
From: WANG, MENGQIU; LIU, LIXIN; ZHANG, TONG; LU, JIA
To: HANGZHOU ZERO ZERO TECHNOLOGY CO., LTD.
Reel/Frame 056206/0829 →
Continuity (4)
Continuation 16448535 · Jun 21, 2019
Continuation 15637954 · Jun 29, 2017
Provisional Application 62479766 · Mar 31, 2017
Related Publication 20210227122A1 · Jul 22, 2021