IP Library Granted Patent US 11,644,839
Granted Patent B2
US 11,644,839 · App. 16/656,431 · Granted May 9, 2023

Systems and methods for generating a real-time map using a movable object

Inventors: Kaiyong Zhao (Shenzhen, CN); Yuewen Ma (Shenzhen, CN); Minglei Zhang (Shenzhen, CN)
Assignee: SZ DJI TECHNOLOGY CO. LTD.
G08G5/0034B64C39/024G01C21/3848G01C21/3852G05D1/0088G05D1/106G06V20/13G08G5/0039B64C2201/127
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,644,839
App. No.
16/656,431
Granted
May 9, 2023
Kind
B2
Abstract

Methods and systems for generating a real-time map of a survey area are provided. The methods and systems may include or be configured to carry out the steps of determining, based on a desired map resolution, a flight path over the survey area for a movable object having at least one image capture device. The methods and systems further may include obtaining images of the survey area captured by the at least one image capture device as the movable object travels along the flight path, and processing the images to generate the real-time map of the survey area with the desired map resolution.

Claims (78)

1. A method of generating a real-time map of a survey area, the method comprising:

determining, based on a desired map resolution selected by a user, a flight path over the survey area for a movable object having at least one image capture device prior to flight;

obtaining images of the survey area captured by the at least one image capture device as the movable object travels along the flight path;

processing the images to generate the real-time snap of the survey area with the desired map resolution;

identifying a missing region in the real-time map of the survey area;

modifying the flight path to capture images corresponding to at least a portion of the missing region;

obtaining sensing data from one or more sensors of the movable object, the sensing data comprising position information or attitude information of the movable object; and

determining camera pose information associated with each image captured by the at least one image capture device based at least in part on the obtained sensing data,

wherein determining the flight path comprises:

determining, based on the desired map resolution, one or more of a height of the movable object, a speed of the movable object, a geographic distance between way points of the flight path, or a frequency for capturing images, and

wherein processing the images to generate the real-time map comprises:

generating a point cloud from each of the images;

generating a digital elevation model (DEM) of the survey area by filtering each of the generated point clouds;

rectifying each of the images based at least in part on the DEM and camera pose information associated with each of the images to generate rectified images;

generating orthoimages with geographic coordinates based on the rectified images; and

stitching the orthoimages to create the real-time map of the survey area.

2. The method of claim 1 , wherein the method is performed by at least one processor on the movable object.

3. The method of claim 1 , wherein the method is performed at a remote terminal configured to receive the images of the survey area from the movable object.

4. The method of claim 1 , wherein determining the flight path comprises:

determining, based on the desired map resolution, the height of the movable object, the speed of the movable object, the geographic distance between way points of the flight path, and the frequency for capturing images.

5. The method of claim 1 , wherein determining the flight path comprises:

designating the survey area based on one or more of existing map data or existing image data, the existing map data or existing image data having a resolution lower than the desired map resolution; and

identifying one or more way points along the flight path over the survey area.

6. The method of claim 1 , wherein the sensing data comprises position or attitude information of the at least one image capture device.

7. The method of claim 1 , further comprising:

identifying an obstacle that prevents the at least one image capture device from capturing images corresponding to a portion of the survey area; and

modifying the flight path to allow the at least one image capture device to capture the images corresponding to that portion of the survey area.

8. The method of claim 1 , further comprising:

identifying an obstacle in the flight path of the movable object; and

modifying the flight path for the movable object to avoid the obstacle.

9. The method of claim 1 , further comprising displaying the generated real-time map of the survey area on a remote controller.

10. The method of claim 1 , further comprising generating navigation instructions on the generated real-time map of the survey area.

11. The method of claim 1 , further comprising incorporating the generated real-time map of the survey area into an existing map.

12. A system for generating a real-time map of a survey area using a movable object with at least one image capture device, the system comprising:

a memory having instructions stored therein; and

a controller including one or more processors configured to execute the instructions to:

determine, based on a desired map resolution selected by a user, a flight path over the survey area for the movable object prior to flight;

obtain images of the survey area captured by the at least one image capture device as the movable object travels along the flight path;

process the images to generate the real-time map of the survey area with the desired map resolution;

identify a missing region in the real-time map of the survey area;

modify the flight path to capture images corresponding to at least a portion of the missing region;

obtain sensing data from one or more sensors of the movable object, the sensing data comprising position information or attitude information of the movable object; and

determine camera pose information associated with each image captured by the at least one image capture device based at least in part on the obtained sensing data,

wherein determining the flight path comprises:

determining, based on the desired map resolution, one or more of a height of the movable object, a speed of the movable object, a geographic distance between way points of the flight path, or a frequency for capturing images, and

wherein processing the images to generate the real-time map comprises:

generating a point cloud from each of the images;

generating a digital elevation model (DEM) of the survey area by filtering each of the generated point clouds;

rectifying each of the images based at least in part on the DEM and camera pose information associated with each of the images to generate rectified images;

generating orthoimages with geographic coordinates based on the rectified images; and

stitching the orthoimages to create the real-time map of the survey area.

13. The system of claim 12 , wherein the one or more processors are further configured to, when determining the flight path:

determining, based on the desired map resolution, the height of the movable object, the speed of the movable object, the geographic distance between way points of the flight path, and the frequency for capturing images.

14. An unmanned aerial vehicle (UAV), comprising:

a propulsion device;

a communication device;

at least one image capture device;

a power storage device configured to power the propulsion device and the communication device;

a memory storing instructions; and

a controller in communication with the communication device and configured to control the UAV to generate a real-time map, the controller comprising one or more processors configured to execute the instructions to:

identify a survey area;

determine, based on a desired map resolution selected by a user, a flight path over the survey area for the UAV prior to flight;

obtain images of the survey area captured by the at least one image capture device as the UAV travels along the flight path;

process the images to generate the real-time map of the survey area with the desired map resolution;

identify a missing region in the real-time map of the survey area;

modify the flight path to capture images corresponding to at least a portion of the missing region;

obtain sensing data from one or more sensors of the movable object, the sensing data comprising position information or attitude information of the movable object; and

determine camera pose information associated with each image captured by the at least one image capture device based at least in part on the obtained sensing data,

wherein determining the flight path comprises:

determining, based on the desired map resolution, one or more of a height of the movable object, a speed of the movable object, a geographic distance between way points of the flight path, or a frequency for capturing images, and

wherein processing the images to generate the real-time map comprises:

generating a point cloud from each of the images;

generating a digital elevation model (DEM) of the survey area by filtering each of the generated point clouds;

rectifying each of the images based at least in part on the DEM and camera pose information associated with each of the images to generate rectified images;

generating orthoimages with geographic coordinates based on the rectified images; and

stitching the orthoimages to create the real-time map of the survey area.

15. The UAV of claim 14 , wherein the one or more processors are further configured to, when determining the flight path, execute the instructions to:

determining, based on the desired map resolution, the height of the movable object, the speed of the movable object, the geographic distance between way points of the flight path, and the frequency for capturing images.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 17, 2019
From: ZHAO, KAIYONG; MA, YUEWEN; ZHANG, MINGLEI
To: SZ DJI TECHNOLOGY CO. LTD.
Reel/Frame 050754/0749 →
Continuity (2)
Continuation PCTCN2017082230 · Apr 27, 2017
Related Publication 20200051443A1 · Feb 13, 2020
Cited By (1)
US 12,608,020