IP Library Granted Patent US 12,276,500
Granted Patent B2
US 12,276,500 · App. 18/351,287 · Granted Apr 15, 2025

Control point identification system

Inventors: Bernard J. Michini (San Francisco, CA); Brett Michael Bethke (Millbrae, CA); Hui Li (San Francisco, CA)
Assignee: Skydio, Inc.
G01C15/02G01C11/02B64U2101/30
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,276,500
App. No.
18/351,287
Granted
Apr 15, 2025
Kind
B2
Abstract

Aerial images captured by a first device are received. A landmark is identified in an aerial image of the aerial images. A determination is made, based on data in a data store, that the landmark is identified as a control point. Precise location information is associated with the control point in the data store. The precise location information indicates a location of a second device at a time that a third device that is different from the second device captured a prior image that includes the landmark. Imagery are generated using photogrammetry software based on the aerial image and the precise location information.

Claims (56)

1. A method, comprising:

receiving vehicle data including precise locations of a vehicle and corresponding location timestamps indicative of times that the vehicle was at the precise locations;

receiving image data including multiple images of the vehicle, corresponding locations of a capture device when the capture device captured each of the multiple images, and corresponding capture timestamps that indicates a time that each of the multiple images was captured;

identifying a landmark in one of the multiple images;

identifying the landmark as a control point;

correlating the vehicle data with the image data to identify precise location information;

associating, in a data store, the control point with at least a portion of the precise location information;

receiving aerial images captured by a first device;

identifying the landmark in an aerial image of the aerial images; and

determining, based on data in the data store, that the landmark is identified as the control point.

2. The method of claim 1 , wherein determining, based on the data in the data store, that the landmark is identified as the control point comprises:

determining that the landmark is identified as the control point based on a match between the precise location information and location information of the first device at a time that the first device captured the aerial image.

3. The method of claim 2 , wherein the precise location information and the location information of the first device are determined to match responsive to a determination that the precise location information and the location information are within a threshold distance.

4. The method of claim 2 , wherein the precise location information and the location information of the first device are determined to match responsive to a determination that a timestamp associated with the precise location information in the data store is within a threshold time period that the first device captured the aerial image.

5. The method of claim 1 , further comprising:

generating imagery using photogrammetry software based on the aerial image and the precise location information,

wherein generating the imagery comprises:

mapping the aerial image to a real-world coordinate frame to obtain a mapped image; and

stitching the mapped image with at least one other mapped image associated with the aerial images.

6. The method of claim 1 , wherein the landmark is identified in the one of the images using a machine learning algorithm trained to identify particular categories of landmarks.

7. The method of claim 1 , wherein the landmark is identified as the control point based on a hierarchy of preferences with respect to selecting landmarks as control points.

8. The method of claim 7 , wherein the hierarchy of the preferences includes a rule preferring two-dimensional landmarks over three-dimensional landmarks.

9. The method of claim 7 , wherein the hierarchy of the preferences includes a rule preferring circular landmarks over irregularly shaped landmarks.

10. The method of claim 7 , wherein the hierarchy of the preferences includes a rule preferring high contrast landmarks over low contrast landmarks.

11. The method of claim 1 , wherein the one of the landmark is identified as the control point based on a score calculated using at least two criteria selected from a dimensionality of the landmark, a shape of the landmark, a contrast of the landmark, whether the landmark moves, and landmark image size.

12. A system, comprising:

a memory; and

a processor, the processor configured to execute instructions stored in the memory to:

receive vehicle data including precise locations of a vehicle and corresponding location timestamps indicative of times that the vehicle was at the precise locations;

receive image data including multiple images of the vehicle, corresponding locations of a capture device when the capture device captured each of the multiple images, and corresponding capture timestamps that indicates a time that each of the multiple images was captured;

identify a landmark in one of multiple received images;

identify the landmark as a control point;

correlate the vehicle data with the image data to identify precise location information;

associate, in a data store, the control point with at least a portion of the precise location information;

identify the landmark in an aerial image; and

determine that the landmark is identified as the control point.

13. The system of claim 12 , wherein the landmark is identified in the aerial image using a machine learning algorithm trained to identify particular categories of landmarks.

14. The system of claim 13 , wherein the particular categories of landmarks include at least a subset of manhole covers, lampposts, sidewalk corners, fire hydrants, roof edges, gutter edges, chimneys, telephone poles, driveways, air conditioning units, swamp coolers, and tops of light poles.

15. The system of claim 12 , wherein to determine that the landmark is identified as the control point comprises instructions to:

determine that the landmark is identified as the control point by matching the precise location information to location information associated with a device at a time that the device captured the aerial image.

16. The system of claim 15 , wherein the precise location information and the location information of the device are determined to match responsive to a determination that the precise location information and the location information are within a threshold distance.

17. The system of claim 12 , wherein the processor is further configured to:

generate a three-dimensional model based on the aerial image and the precise location information.

18. A non-transitory computer readable medium storing instructions operable to cause one or more processors to perform operations to:

receive vehicle data including precise locations of a vehicle and corresponding location timestamps indicative of times that the vehicle was at the precise locations;

receive image data including multiple images of the vehicle, corresponding locations of a capture device when the capture device captured each of the multiple images, and corresponding capture timestamps that indicates a time that each of the multiple images was captured;

identify a landmark in one of multiple received images;

identify the landmark as a control point;

correlate the vehicle data with the image data to identify precise location information;

associate, in a data store, the control point with at least a portion of the precise location information;

receive aerial images from a first device;

identify the landmark in an aerial image of the aerial images; and

determine that the landmark is identified as the control point.

19. The non-transitory computer readable medium of claim 18 , wherein to determine that the landmark is identified as the control point comprise to:

determine that the landmark is identified as the control point by matching the precise location information to location information associated with a device at a time that the device captured the aerial image.

20. The non-transitory computer readable medium of claim 19 , wherein the precise location information and the location information of the first device are determined to match responsive to a determination that a timestamp associated with the precise location information in the data store is within a threshold time period of the time that the first device captured the aerial image.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 13, 2023
From: MICHINI, BERNARD J.; BETHKE, BRETT MICHAEL; LI, HUI
To: UNMANNED INNOVATION INC.
Reel/Frame 064240/0425 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 13, 2023
From: UNMANNED INNOVATION INC.
To: AIRWARE, LLC
Reel/Frame 064240/0462 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 13, 2023
From: AIRWARE, LLC
To: SKYDIO, INC.
Reel/Frame 064240/0537 →
Continuity (4)
Continuation 17478220 · Sep 17, 2021
Continuation 15396096 · Dec 30, 2016
Provisional Application 62273822 · Dec 31, 2015
Related Publication 20230358537A1 · Nov 9, 2023
References Cited (24)
US 6437708B1 · Brouwer · 2002 [cited by examiner]
US 8315794B1 · Strelow et al. · 2012 [cited by applicant]
US 9052571B1 · Lapstun et al. · 2015 [cited by applicant]
US 9087451B1 · Jarrell · 2015 [cited by applicant]
US 9415869B1 · Chan et al. · 2016 [cited by applicant]
US 9529360B1 · Melamed et al. · 2016 [cited by applicant]
US 11150089B2 · Michini et al. · 2021 [cited by applicant]
US 20140316616A1 · Kugelmass · 2014 [cited by examiner]
US 20150032295A1 · Stark et al. · 2015 [cited by applicant]
US 20150148988A1 · Fleck · 2015 [cited by applicant]
US 20150269438A1 · Samarasekera · 2015 [cited by examiner]
US 20150321758A1 · Sarna, II · 2015 [cited by applicant]
US 20160004795A1 · Novak · 2016 [cited by examiner]
US 20160023760A1 · Goodrich · 2016 [cited by applicant]
US 20170109577A1 · Wang · 2017 [cited by examiner]
US 20170234966A1 · Naguib et al. · 2017 [cited by applicant]
US 20180155023A1 · Choi et al. · 2018 [cited by applicant]
US 20180305012A1 · Ichihara · 2018 [cited by applicant]
CN 204854730U · 2015 [cited by examiner]
CN 107615358A · 2018 [cited by applicant]
Douterloigne, Koen & Gautama, Sidharta & Philips, Wilfried. (2010). On the accuracy of 3D landscapes from UAV image data. 589-592. 10.1109/IGARSS.2010.5651391. (Year: 2010). [cited by examiner]
S. Minaelan, “Vision-Based Target Detection and Localization via a Team of Cooperative UAV and UGVs”, 2015 (Year: 2015). [cited by applicant]
Jayatilleke, Lalindra et al., “Landmark-Based Localization For Unmanned Aerial Vehicles,” 2013 IEEE International Systems Conference, 4 pages, Apr. 15-18, 2013,. [cited by applicant]
Sharma, Rajnikant, “Observability Based Control For Cooperative Localization,” 2014 International Conference on Unmanned Aircraft Systems, pp. 134-139, May 27-30, 2014,. [cited by applicant]