IP Library Granted Patent US 12,217,181
Granted Patent B2
US 12,217,181 · App. 18/520,014 · Granted Feb 4, 2025

Visual observer for unmanned aerial vehicles

Inventors: Eric Miller (Granville, OH); Jacob Daniel Phillips (Boston, MA)
Assignee: SKYDIO, INC.
G06N3/08B64U20/87G05D1/0094G05D1/1064G05D1/12G06V10/25G06V10/764G06V10/80G06V10/82G06V20/17G06V20/52H04N23/695B64U2101/30B64U2201/20
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,217,181
App. No.
18/520,014
Granted
Feb 4, 2025
Kind
B2
Abstract

In some examples, a first device includes multiple fixed first cameras and a movable second camera. A processor is configured to receive, from at least one of the fixed first cameras, a plurality of first images of an airspace corresponding to an area of operation of an unmanned aerial vehicle, and detect, based at least on the first images, a candidate object approaching or within the airspace. Based on detecting the candidate object, the processor controls a movable second camera to direct a field of view of the movable second camera toward the candidate object. Based on one or more second images from the movable second camera captured at a first location and one or more third images from a third camera captured at a second location, the processor may determine that the candidate object is an object of interest and perform at least one action.

Claims (65)

1. A system comprising:

a first device including a plurality of cameras, the plurality of cameras including multiple fixed first cameras, each fixed first camera having a respective field of view (FOV) pointing in a different direction from at least one other fixed first camera of the multiple fixed first cameras, the plurality of cameras further including a movable second camera mounted between the respective FOVs of two of the fixed first cameras; and

at least one processor associated with the first device, the at least one processor configured by executable instructions to:

receive, from at least one of the fixed first cameras of the multiple fixed first cameras, a plurality of first images of an airspace corresponding to an area of operation of a first unmanned aerial vehicle (UAV);

detect, based at least on the plurality of first images, a candidate object approaching or within the airspace;

based on detecting the candidate object, control a movable second camera to direct an FOV of the movable second camera toward the candidate object, wherein the moveable second camera has an optical magnification that is greater than optical magnifications of the multiple fixed first cameras;

determine, based on one or more second images from the movable second camera captured at a first location and one or more third images from a third camera captured at a second location that the candidate object is an object of interest; and

perform at least one action based on determining that the candidate object is the object of interest.

2. The system as recited in claim 1 , the at least one processor further configured by the executable instructions to at least one of:

perform the at least one action by sending an instruction to cause the first UAV to perform an action based on the object of interest; or

perform the at least one action by sending a notification to a computing device to provide, to the computing device, information related to the object of interest.

3. The system as recited in claim 1 , the at least one processor further configured by the executable instructions to check at least one of received communications or sensor information from at least one other sensor to improve or lower a confidence in determining that the candidate object is the object of interest.

4. The system as recited in claim 1 , wherein the third camera is associated with a second device, the at least one processor further configured by the executable instructions to:

based on detecting the candidate object, send, to the second device, at least an indication of a location of the candidate object; and

receive, from the second device, at least partially in response to the sending the indication of the location of the candidate object, the one or more images from the third camera.

5. The system as recited in claim 1 , wherein at least one of:

the third camera is associated with a second device, and the second device is a second UAV having the third camera mounted thereon; or

the first device is a second UAV having the plurality of cameras mounted thereon.

6. The system as recited in claim 1 , the at least one processor further configured by the executable instructions to detect the candidate object approaching or within the airspace by using a first computer vision algorithm to detect the candidate object and by using a second computer vision algorithm to determine an indication of whether an output of the first computer vision algorithm is correct.

7. The system as recited in claim 6 , the at least one processor further configured by the executable instructions to use a third computer vision algorithm to determine, based on the one or more images from the movable second camera captured at the first location, and the one or more images from the third camera captured at the second location, that the candidate object is an object of interest.

8. A system comprising:

a first device including a plurality of cameras, the plurality of cameras including multiple fixed first cameras, the plurality of cameras further including a movable second camera; and

at least one processor associated with the first device, the at least one processor configured by executable instructions to:

receive, from at least one of the fixed first cameras of the multiple fixed first cameras, a plurality of first images of an airspace corresponding to an area of operation of a first unmanned aerial vehicle (UAV);

detect, based at least on the plurality of first images, a candidate object approaching or within the airspace;

based on detecting the candidate object, control a movable second camera to direct an FOV of the movable second camera toward the candidate object, wherein the moveable second camera has an optical magnification that is greater than optical magnifications of the multiple fixed first cameras;

determine, based on one or more second images from the movable second camera captured at a first location and one or more third images from a third camera captured at a second location that the candidate object is an object of interest; and

perform at least one action based on determining that the candidate object is the object of interest.

9. The system as recited in claim 8 , the at least one processor further configured by the executable instructions to at least one of:

perform the at least one action by sending an instruction to cause the first UAV to perform an action based on the object of interest; or

perform the at least one action by sending a notification to a computing device to provide, to the computing device, information related to the object of interest.

10. The system as recited in claim 8 , the at least one processor further configured by the executable instructions to check at least one of received communications or sensor information from at least one other sensor to improve or lower a confidence in determining that the candidate object is the object of interest.

11. The system as recited in claim 8 , wherein the third camera is associated with a second device, the at least one processor further configured by the executable instructions to:

based on detecting the candidate object, send, to the second device, at least an indication of a location of the candidate object; and

receive, from the second device, at least partially in response to the sending the indication of the location of the candidate object, the one or more images from the third camera.

12. The system as recited in claim 8 , wherein at least one of:

the third camera is associated with a second device, and the second device is a second UAV having the third camera mounted thereon; or

the first device is a second UAV having the plurality of cameras mounted thereon.

13. The system as recited in claim 8 , the at least one processor further configured by the executable instructions to detect the candidate object approaching or within the airspace by using a first computer vision algorithm to detect the candidate object and by using a second computer vision algorithm to determine an indication of whether an output of the first computer vision algorithm is correct.

14. The system as recited in claim 8 , wherein:

at least some of the multiple fixed first cameras of the first device are arranged radially;

the respective FOVs of the radially arranged fixed first cameras each point in a different direction; and

the movable second camera is mounted on the first device between the FOVs of two of the radially arranged fixed first cameras.

15. A system comprising:

a first device including a plurality of cameras, the plurality of cameras including multiple fixed first cameras, each fixed first camera having a respective field of view (FOV) pointing in a different direction from at least one other fixed first camera of the multiple fixed first cameras, the plurality of cameras further including a movable second camera mounted between the respective FOVs of two of the fixed first cameras; and

at least one processor associated with the first device, the at least one processor configured by executable instructions to:

receive, from at least one of the fixed first cameras of the multiple fixed first cameras, a plurality of first images of an airspace corresponding to an area of operation of a first unmanned aerial vehicle (UAV);

detect, based at least on the plurality of first images, a candidate object approaching or within the airspace;

based on detecting the candidate object, control a movable second camera to direct an FOV of the movable second camera toward the candidate object;

determine, based on one or more second images from the movable second camera captured at a first location and one or more third images from a third camera captured at a second location that the candidate object is an object of interest; and

perform at least one action based on determining that the candidate object is the object of interest.

16. The system as recited in claim 15 , the at least one processor further configured by the executable instructions to at least one of:

perform the at least one action by sending an instruction to cause the first UAV to perform an action based on the object of interest; or

perform the at least one action by sending a notification to a computing device to provide, to the computing device, information related to the object of interest.

17. The system as recited in claim 15 , the at least one processor further configured by the executable instructions to check at least one of received communications or sensor information from at least one other sensor to improve or lower a confidence in determining that the candidate object is the object of interest.

18. The system as recited in claim 15 , wherein the third camera is associated with a second device, the at least one processor further configured by the executable instructions to:

based on detecting the candidate object, send, to the second device, at least an indication of a location of the candidate object; and

receive, from the second device, at least partially in response to the sending the indication of the location of the candidate object, the one or more images from the third camera.

19. The system as recited in claim 15 , wherein at least one of:

the third camera is associated with a second device, and the second device is a second UAV having the third camera mounted thereon; or

the first device is a second UAV having the plurality of cameras mounted thereon, and the moveable second camera has an optical magnification that is greater than optical magnifications of the multiple fixed first cameras.

20. The system as recited in claim 15 , wherein:

at least some of the multiple fixed first cameras of the first device are arranged radially;

the respective FOVs of the radially arranged fixed first cameras each point in a different direction; and

the movable second camera is mounted on the first device between the FOVs of two of the radially arranged fixed first cameras.

Assignments (2)
SECURITY INTEREST Recorded Dec 5, 2024
From: SKYDIO, INC.
To: ACQUIOM AGENCY SERVICES LLC
Reel/Frame 069516/0452 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 27, 2023
From: MILLER, ERIC; PHILLIPS, JACOB DANIEL
To: SKYDIO, INC.
Reel/Frame 065670/0467 →
Continuity (3)
Continuation 17314885 · May 7, 2021
Provisional Application 63021329 · May 7, 2020
Related Publication 20240101255A1 · Mar 28, 2024
References Cited (7)
US 10554950B1 · Curlander et al. · 2020 [cited by applicant]
US 11170524B1 · Mishra et al. · 2021 [cited by applicant]
US 11249182B2 · Luders et al. · 2022 [cited by applicant]
US 20160078759A1 · Nerayoff et al. · 2016 [cited by applicant]
CN 105704437B · 2021 [cited by applicant]
DE 102016011492 · 2017 [cited by examiner]
WO WO2021166008 · 2021 [cited by examiner]