IP Library › Granted Patent US 10,949,679
Granted Patent B2
US 10,949,679 · App. 16/126,509 · Granted Mar 16, 2021

Nighttime sensing

Inventors: Lucian Ion (Mountain View, CA); Daniel E. Potter (San Jose, CA); Evan C. Cull (Sunnyvale, CA); Xiaofeng Tang (San Jose, CA)
Assignee: Apple Inc.
G06K9/00791G05D1/0214G05D1/0246G06K9/00805G06K9/2018G06K9/2027G06K9/3241G06K9/6267H04N5/2258H04N5/2354H04N5/23218H04N5/332G05D2201/0213
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,949,679
App. No.
16/126,509
Granted
Mar 16, 2021
Kind
B2
Abstract

Systems and methods for night vision combining sensor image types. Some implementations may include obtaining a long wave infrared image from a long wave infrared sensor; detecting an object in the long wave infrared image; identifying a region of interest associated with the object; adjusting a control parameter of a near infrared sensor based on data associated with the region of interest; obtaining a near infrared image captured using the adjusted control parameter of the near infrared sensor; and determining a classification of the object based on data of the near infrared image associated with the region of interest.

Claims (79)

1. A system, comprising:

a long wave infrared sensor;

a near infrared sensor;

a visible spectrum sensor;

a data processing apparatus; and

a data storage device storing instructions executable by the data processing apparatus that upon execution by the data processing apparatus cause the data processing apparatus to perform operations comprising:

obtaining a long wave infrared image from the long wave infrared sensor,

detecting an object in the long wave infrared image,

identifying a region of interest associated with the object,

adjusting a control parameter of the near infrared sensor based on data associated with the region of interest,

adjusting a control parameter of the visible spectrum sensor based on data associated with the region of interest,

obtaining a near infrared image captured using the adjusted control parameter of the near infrared sensor,

obtaining a visible spectrum image from the visible spectrum sensor that is captured using the adjusted control parameter of the visible spectrum sensor, and

determining a classification of the object based on data of the near infrared image associated with the region of interest and based on data of the visible spectrum image associated with the region of interest.

2. The system of claim 1 , further comprising:

a near infrared illuminator;

wherein the operations comprise adjusting a near infrared illuminator control parameter based on data associated with the region of interest; and

wherein the near infrared image is captured using the adjusted near infrared illuminator control parameter.

3. The system of claim 2 , wherein the near infrared illuminator control parameter is a brightness.

4. The system of claim 2 , wherein the near infrared illuminator control parameter is field of illumination.

5. The system of claim 1 , wherein the control parameter of the near infrared sensor is an integration time.

6. The system of claim 1 , wherein the control parameter of the near infrared sensor is an aperture size.

7. The system of claim 1 , wherein the control parameter of the near infrared sensor is a filter selection.

8. The system of claim 1 , wherein the control parameter of the near infrared sensor is an amplification gain.

9. The system of claim 1 , further comprising:

a visible spectrum illuminator;

wherein the operations comprise adjusting a visible spectrum illuminator control parameter based on data associated with the region of interest; and

wherein the visible spectrum image is captured using the adjusted visible spectrum illuminator control parameter.

10. The system of claim 1 , wherein the operations further comprise:

adjusting a computational control parameter based on data associated with the region of interest; and

wherein the classification of the object is determined using the computational control parameter.

11. A method, comprising:

obtaining a long wave infrared image from a long wave infrared sensor;

detecting an object in the long wave infrared image;

identifying a region of interest associated with the object;

adjusting a control parameter of a near infrared sensor based on data associated with the region of interest;

adjusting a visible spectrum sensor control parameter based on data associated with the region of interest;

obtaining a near infrared image captured using the adjusted control parameter of the near infrared sensor;

obtaining a visible spectrum image from the visible spectrum sensor that is captured using the adjusted control parameter of the visible spectrum sensor; and

determining a classification of the object based on data of the near infrared image associated with the region of interest and based on data of the visible spectrum image associated with the region of interest.

12. The method of claim 11 , further comprising:

adjusting a visible spectrum illuminator control parameter based on data associated with the region of interest; and

wherein the visible spectrum image is captured using the adjusted visible spectrum illuminator control parameter.

13. The method of claim 11 , further comprising:

adjusting a near infrared illuminator control parameter based on data associated with the region of interest; and

wherein the near infrared image is captured using the adjusted near infrared illuminator control parameter.

14. The method of claim 11 , wherein the near infrared image has a higher resolution than the long wave infrared image.

15. The method of claim 11 , further comprising:

adjusting a computational control parameter based on data associated with the region of interest; and

wherein the classification of the object is determined using the computational control parameter.

16. A vehicle, comprising:

a vehicle body;

actuators operable to cause motion of the vehicle body;

a long wave infrared sensor;

a near infrared sensor;

a visible spectrum sensor; and

an automated controller configured to:

obtain a long wave infrared image from the long wave infrared sensor,

detect an object in the long wave infrared image,

identify a region of interest associated with the object,

adjust a control parameter of the near infrared sensor based on data associated with the region of interest,

adjust a control parameter of the visible spectrum sensor based on data associated with the region of interest,

obtain a near infrared image captured using the adjusted control parameter of the near infrared sensor,

obtain a visible spectrum image from the visible spectrum sensor that is captured using the adjusted control parameter of the visible spectrum sensor,

determine a classification of the object based on data of the near infrared image associated with the region of interest and based on data of the visible spectrum image associated with the region of interest,

determine a motion plan based on the classification of the object, and

output commands to the actuators to maneuver the vehicle.

17. The vehicle of claim 16 , further comprising:

a visible spectrum illuminator;

wherein the automated controller is configured to adjust a visible spectrum illuminator control parameter based on data associated with the region of interest; and

wherein the visible spectrum image is captured using the adjusted visible spectrum illuminator control parameter.

18. The vehicle of claim 16 , further comprising:

a near infrared illuminator;

wherein the automated controller is configured to adjust a near infrared illuminator control parameter based on data associated with the region of interest; and

wherein the near infrared image is captured using the adjusted near infrared illuminator control parameter.

19. The vehicle of claim 16 , wherein the near infrared image has a higher resolution than the long wave infrared image.

20. The vehicle of claim 16 , wherein the automated controller is further configured to:

adjust a computational control parameter based on data associated with the region of interest; and

wherein the classification of the object is determined using the computational control parameter.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 11, 2018
From: ION, LUCIAN; POTTER, DANIEL E.; CULL, EVAN C.; TANG, XIAOFENG
To: APPLE INC.
Reel/Frame 046836/0858 →
Continuity (2)
Provisional Application 62564654 · Sep 28, 2017
Related Publication 20190095721A1 · Mar 28, 2019