Granted Patent B2

US 11,768,508 · App. 15/042,630 · Granted Sep 26, 2023

Inventors: Hui Li (San Francisco, CA); John Bressler (San Francisco, CA); Volkan Gurel (San Francisco, CA); Bernard J. Michini (San Francisco, CA)

Assignee: Skydio, Inc.

G05D1/101G01C21/1656G05D1/0038H04N5/232H04N5/23203H04N5/23222H04N5/33H04N7/185H04N7/188B64C39/024B64C2201/123B64C2201/127

Abstract

Claims (66)

1. A computerized method performed by an unmanned aerial vehicle (UAV) comprising one or more processors, the method comprising:

obtaining information describing a flight plan for implementation, the flight plan including one or more way points that trigger a camera of the UAV to capture images;

recording flight log data including geospatial locations of the UAV and a time at which the camera was triggered to capture the images;

determining a geospatial location for at least one image of the images by:

determining a time offset between a timestamp associated with the at least one image and a timestamp included in the flight log data,

identifying, based on the time offset, a subset of the geospatial locations of the UAV at timestamps adjacent to the timestamp included in the flight log data that corresponds to the timestamp associated with the at least one image, and

determining, using the subset of the geospatial locations of the UAV, the geospatial location for the at least one image; and

providing, to a user device in communication with the UAV, information indicating that the camera captured the images, wherein the user device monitors the UAV during implementation of the flight plan.

2. The method of claim 1 , further comprising:

obtaining information identifying a ground sampling distance, the ground sampling distance indicating a minimum acceptable number of image pixels per distance;

determining, based on the ground sampling distance and information describing capabilities of the camera, a minimum altitude of the UAV that will allow one or more of the images to include the ground sampling distance;

obtaining an altitude of the UAV while the one or more of the images was captured; and

determining that the obtained altitude is greater than the minimum altitude, and determining to recapture the one or more of the images.

3. The method of claim 1 , further comprising:

determining, that while one or more of the images was captured, the UAV moved greater than a threshold distance; and

determining to recapture the one or more of the images.

4. The method of claim 1 , further comprising:

determining one or more quality scores associated with one or more of the images, wherein a given quality score comprises a measure of a respective measures of quality of the one or more of the images; and

determining that at least one quality score is less than an acceptable threshold, and determining to recapture the one or more of the images.

5. The method of claim 4 , wherein a measure of quality comprises a measure of sharpness of the one or more of the images, a measure of blurriness of the one or more of the images, a measure of overlap between the one or more of the images and one or more prior images, a measure of exposure of the one or more of the images, a measure of contrast of the one or more of the images, or any combination thereof.

6. The method of claim 1 , further comprising:

generating a reduced quality version of one or more of the images; and

providing, to the user device, the reduced quality version of the one or more of the images for presentation on the user device.

7. The method of claim 6 , further comprising:

receiving, from the user device, information describing that the one or more of the images is to be recaptured,

wherein a user of the user device indicates that the one or more of the images is unacceptable, or the user device determines that at least one quality score associated with the one or more of the images is below an acceptable threshold.

8. The method of claim 6 , further comprising:

receiving, from the user device, a request for the one or more of the images as captured by the camera;

providing the one or more of the images to the user device; and

receiving, from the user device, information indicating recapture of the one or more of the images.

9. The method of claim 1 , wherein a trigger comprises a location at which the UAV is to capture one or more of the images, a time at which the UAV is to capture one or more of the images, a determination to recapture one or more of the images, or receipt of information from the user device indicating that the UAV is to capture, or recapture, one or more of the images.

10. The method of claim 1 , further comprising:

in response to providing information indicating that the camera is to capture one or more of the images, obtaining location information of the UAV, and associating the location information with one or more timestamps; and

providing, at a completion of the flight plan to an outside system, the one or more of the images, associated timestamps, and associated location information for processing by the outside system.

11. A computer-implemented method comprising:

obtaining images and log data associated with an unmanned aerial vehicle (UAV) implementing a flight plan, each image being captured by a camera of the UAV and including metadata, generated by the camera, indicating a timestamp at which the camera captured the image, and the log data including image capture timestamps associated with the UAV receiving information, from the camera, identifying that images were captured along with timestamps associated with determined location information of the UAV and inertial navigation system (INS) data of the UAV;

determining a time series of metadata timestamps and a time series of image capture timestamps;

determining an optimal time offset between the time series of metadata timestamps and the time series of image capture timestamps, the optimal time offset indicating, for one or more images, a difference in value between respective metadata timestamps and corresponding image capture timestamps;

determining, using the optimal time offset for each obtained image, estimated location information of the UAV and INS data of the UAV when the image was captured; and

generating geo-rectified imagery from the obtained images and associated location information of the UAV and INS data of the UAV.

12. The computer-implemented method of claim 11 , wherein determining an optimal time offset comprises:

assigning an initial time offset, and performing a correlation between the time series of metadata timestamps and the time series of image capture timestamps using the initial time offset; and

updating the initial time offset by one or more values, and performing the correlation, for each updated time offset; and

assigning the updated time offset that results in a greater correlation as the optimal time offset.

13. The computer-implemented method of claim 11 , wherein determining an optimal time offset comprises:

for one or more time offsets applied to the time series of metadata timestamps, determining the optimal time offset as the time off set that matches a greatest number of metadata timestamps to image captured timestamps.

14. An unmanned aerial vehicle (UAV) system comprising one or more computers and one or more non-transitory computer storage media storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the operations comprising:

receiving a flight plan, the flight plan including one or more way points;

navigating the UAV along the one or more way points according to the received flight plan;

at least at one of the way points, triggering a camera to capture digital images;

recording flight log data comprising at least, geospatial location of the UAV and a time at which the camera was triggered; and

determining a geospatial location for one or more of the digital images by determining an optimal time offset between timestamps associated with the one or more digital images and timestamps included in the flight log data, identifying, using the optimal time offset and a particular timestamp associated with a particular digital image, geospatial locations of the UAV at timestamps adjacent to a timestamp included in the flight log data that corresponds to the particular timestamp, and determining, using the identified geospatial locations, an estimated geospatial location for the particular digital image.

15. The system of claim 14 , wherein the operations further comprise:

obtaining information identifying a ground sampling distance, the ground sampling distance indicating a minimum acceptable number of image pixels per distance;

determining, using the ground sampling distance and information describing capabilities of the camera, a minimum altitude of the UAV that will allow captured images to include the ground sampling distance;

obtaining an altitude of the UAV while the digital image was captured; and

determining that the obtained altitude is greater than the minimum altitude, and determining that the digital image is to be re-captured.

16. The system of claim 14 , wherein a trigger comprises a location at which the UAV is to capture an image, a time at which the UAV is to capture an image, a determination to re-capture an image, or receipt of information from a user device in wireless communication with the UAV indicating that the UAV is to capture, or re-capture, an image.

17. A system comprising one or more computers and one or more non-transitory computer storage media storing instructions that are operable, when executed by the one or more computers, to cause the one or more computer to perform the operations comprising:

receiving a flight plan including one or more waypoints for navigating an unmanned aerial vehicle (UAV) along the one or more waypoints according to the flight plan, wherein the waypoints include associated information to trigger a camera to capture one or more digital images;

transmitting the flight plan to a UAV computer system;

receiving UAV flight log data comprising at least, geospatial location of the UAV and a timestamp at which the camera was triggered;

receiving digital images taken by the camera, wherein the digital images have an associated timestamp when the image was taken; and

determining a geospatial location for one or more of the digital images by determining an optimal time offset between timestamps associated with the one or more digital images and timestamps included in the flight log data, the optimal time offset indicating a difference in timestamp value between timestamps associated with digital images and corresponding timestamps included in the flight log data, identifying, using the optimal time offset and a particular timestamp associated with a particular digital image, geospatial locations of the UAV at timestamps adjacent to a timestamp included in the flight log data that corresponds to the particular timestamp, and determining, using the identified geospatial locations, an estimated geospatial location for the particular digital image.

18. The system of claim 17 , wherein the identified geospatial locations are geospatial locations at a closest timestamp prior to the timestamp included in the flight log data that corresponds to the particular timestamp and a closest timestamp subsequent to the timestamp included in the flight log data that corresponds to the particular timestamp.

19. The system of claim 17 , wherein the geospatial locations of the UAV are associated with respective periodic timestamps.

Assignments (9)

Continuity (2)