IP Library Granted Patent US 11,861,896
Granted Patent B1
US 11,861,896 · App. 17/707,841 · Granted Jan 2, 2024

Autonomous aerial navigation in low-light and no-light conditions

Inventors: Samuel Shenghung Wang (Mountain View, CA); Vladimir Nekrasov (Adelaide, AU); Ryan David Kennedy (Redwood City, CA); Gareth Benoit Cross (Mountain View, CA); Peter Benjamin Henry (San Francisco, CA); Kristen Marie Holtz (Menlo Park, CA); Hayk Martirosyan (San Francisco, CA); Abraham Galton Bachrach (Redwood City, CA); Adam Parker Bry (Redwood City, CA)
Assignee: Skydio, Inc.
G06V20/17B64C39/024G05D1/101G06T3/4038G06T5/008G06V10/30G06V10/60G06V10/82H04N5/33B64U2101/30B64U2201/10G06T2207/10024G06T2207/10032G06T2207/20024G06T2207/20081G06T2207/20084G06T2207/20182G06T2207/30252
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,861,896
App. No.
17/707,841
Granted
Jan 2, 2024
Kind
B1
Abstract

Autonomous aerial navigation in low-light and no-light conditions includes using night mode obstacle avoidance intelligence, training, and mechanisms for vision-based unmanned aerial vehicle (UAV) navigation to enable autonomous flight operations of a UAV in low-light and no-light environments using infrared data.

Claims (42)

1. An apparatus, comprising:

a memory configured to store instructions for training a learning model for use with unmanned aerial vehicle navigation; and

a processor configured to execute the instructions stored in the memory to:

produce a first image including infrared data from an infrared light onboard an unmanned aerial vehicle by simulating a reflection of the infrared data to determine a simulated infrared illumination range within an environment depicted by a first copy of input image data;

perform range-based darkening against a second copy of the input image data to produce a second image including darkened RGB color data;

combine the first image and the second image to produce a combined image; and

train the learning model based on the combined image.

2. The apparatus of claim 1 , wherein the simulated infrared illumination range indicates how the reflection of the infrared data interacts with exposure features of an onboard camera of the unmanned aerial vehicle.

3. The apparatus of claim 1 , wherein, to perform the range-based darkening, the processor is configured to execute the instructions to:

applying a darkening filter to one or more portions of the second copy of the input image data to darken RGB values within the one or more portions.

4. The apparatus of claim 3 , wherein the one or more portions are determined based on an expected range of infrared illumination.

5. The apparatus of claim 1 , wherein the processor is further configured to execute the instructions to:

introduce camera noise within the combined image prior to using the combined image for training the learning model.

6. The apparatus of claim 5 , wherein the combined image including the camera noise represents image data that an onboard camera of an unmanned aerial vehicle is configured to capture while the unmanned aerial vehicle is in a night mode configuration.

7. The apparatus of claim 1 , wherein the input image data includes one or more images captured using an infrared filter of an onboard camera of an unmanned aerial vehicle.

8. The apparatus of claim 7 , wherein the trained learning model is used by the unmanned aerial vehicle while the unmanned aerial vehicle is a night mode configuration.

9. A method, comprising:

producing a first image based on a simulated infrared illumination range determined within an environment depicted by a first copy of input image data by simulating a reflection of infrared data from an infrared light onboard an unmanned aerial vehicle;

producing a second image based on a range-based darkening performed against a second copy of the input image data;

producing a combined image by blending the first image and the second image;

training a learning model based on the combined image; and

providing the trained learning model for use by one or more unmanned aerial vehicles while the one or more unmanned aerial vehicles navigate in a night mode configuration.

10. The method of claim 9 , wherein the

one or more unmanned aerial vehicles include the unmanned aerial vehicle from which the infrared data is obtained.

11. The method of claim 9 , wherein producing the second image comprises:

darkening RGB values within the second copy of the input image data using a darkening filter.

12. The method of claim 9 , comprising:

prior to using the combined image for training the learning model, introducing camera noise into the combined image to cause the combined image to represent image data that one or more cameras of the one or more unmanned aerial vehicles are configured to capture.

13. The method of claim 9 , comprising:

transmitting, based on changes at a server at which the learning model is trained, an update to the learning model to the one or more unmanned aerial vehicles.

14. The method of claim 9 , wherein the combined image includes infrared data of the first image and darkened RGB color data of the second image.

15. The method of claim 9 , wherein the input image data includes one or more images captured by a camera of the unmanned aerial vehicle and processed to remove infrared data.

16. A non-transitory computer storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

blending a first image produced based on an infrared reflection mask simulation performed against a first copy of input image data and a second image produced based on a range-based darkening performed against a second copy of the input image data to produce a combined image, wherein performing the infrared reflection mask simulation includes simulating a reflection of infrared data to determine a simulated infrared illumination range within an environment depicted by the first copy of input image data;

training a learning model using the combined image; and

providing the trained learning model for use by an unmanned aerial vehicle while the unmanned aerial vehicle navigates while in a night mode configuration.

17. The non-transitory computer storage medium of claim 16 , the operations comprising:

performing the range-based darkening by applying a darkening filter to one or more portions of the second copy of the input image data to darken RGB values within the one or more portions.

18. The non-transitory computer storage medium of claim 16 , the operations comprising:

preparing the combined image for use in training the learning model by augmenting the combined image with camera noise.

19. The non-transitory computer storage medium of claim 16 , wherein the combined image includes infrared data of the first image and darkened RGB color data of the second image.

20. The non-transitory computer storage medium of claim 16 , wherein the infrared data is from an infrared light onboard the unmanned aerial vehicle.

Assignments (2)
SECURITY INTEREST Recorded Dec 5, 2024
From: SKYDIO, INC.
To: ACQUIOM AGENCY SERVICES LLC
Reel/Frame 069516/0452 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 14, 2022
From: WANG, SAMUEL SHENGHUNG; NEKRASOV, VLADIMIR; KENNEDY, RYAN DAVID; CROSS, GARETH BENOIT; HENRY, PETER BENJAMIN; HOLTZ, KRISTEN MARIE; MARTIROSYAN, HAYK; BACHRACH, ABRAHAM GALTON; BRY, ADAM PARKER
To: SKYDIO, INC.
Reel/Frame 060191/0609 →
Continuity (1)
Provisional Application 63168827 · Mar 31, 2021
Cited By (1)
US 12,634,586