IP Library Granted Patent US 11,196,916
Granted Patent B2
US 11,196,916 · App. 16/935,931 · Granted Dec 7, 2021

Identification of an object based on identifying portions of the object captured by multiple image sensors having different luminance levels

Inventors: Andreas Wendel (Mountain View, CA); Jeremy Dittmer (Mountain View, CA); Brendan Hermalyn (San Francisco, CA); Benjamin Ingram (Santa Clara, CA)
Assignee: Waymo LLC
H04N5/23218H04N5/2258H04N5/2353H04N5/2355
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,196,916
App. No.
16/935,931
Granted
Dec 7, 2021
Kind
B2
Abstract

Example embodiments relate to multiple operating modes to expand dynamic range. An example embodiment includes a camera system. The camera system may include a first image sensor having a first dynamic range corresponding to a first range of luminance levels in a scene. The system may also include a second image sensor having a second dynamic range corresponding to a second range of luminance levels in the scene. The camera system may further include a processor coupled to the first image sensor and the second image sensor. The processor may be configured to execute instructions to identify objects of a first type in a first image of the scene captured by the first image sensor and identify objects of a second object type in a second image of the scene captured by the second image sensor.

Claims (68)

1. A camera system, comprising:

a first image sensor having a first dynamic range corresponding to a first range of luminance levels in a scene;

a second image sensor having a second dynamic range corresponding to a second range of luminance levels in the scene, wherein the second range of luminance levels includes luminance levels that are higher than the first range of luminance levels; and

a processor coupled to the first image sensor and the second image sensor, wherein the processor is configured to execute instructions to:

identify a first portion of an object based on a first image of the scene captured by the first image sensor, wherein the first portion of the object exhibits a first luminance corresponding to the first range of luminance levels;

identify a second portion of the object, different from the first portion, based on a second image of the scene captured by the second image sensor, wherein the second portion of the object exhibits a second luminance corresponding to the second range of luminance levels; and

identify the object based on the identified first and second portions.

2. The camera system of claim 1 ,

wherein the first image sensor has a variable exposure that is adjusted by a controller executing instructions based on illuminance, and

wherein the second image sensor has a fixed exposure.

3. The camera system of claim 2 , wherein adjusting the variable exposure based on luminance comprises adjusting an exposure duration, an aperture size, a gain level, or an ISO sensitivity of the first image sensor.

4. The camera system of claim 2 ,

wherein the variable exposure is adjusted by modifying a shutter speed or gain associated with a shutter on the first image sensor.

5. The camera system of claim 1 , further comprising a third image sensor having a third dynamic range corresponding to a third range of luminance levels in the scene,

wherein the processor is coupled to the third image sensor,

wherein the third range of luminance levels includes at least one luminance level that is lower than the first range of luminance levels, and

wherein the processor is further configured to execute instructions to identify the object based on a third image of the scene captured by the third image sensor.

6. The camera system of claim 5 , wherein the third image sensor has a variable exposure that is adjusted by a controller executing instructions based on illuminance.

7. The camera system of claim 6 , wherein adjusting the variable exposure based on luminance comprises adjusting an exposure duration, shutter speed, an aperture size, or an ISO sensitivity of the third image sensor.

8. The camera system of claim 1 , further comprising:

a third image sensor having the first dynamic range; and

a fourth image sensor having the second dynamic range,

wherein the processor is coupled to the third image sensor and the fourth image sensor, and

wherein the processor is further configured to execute instructions to:

identify objects of a first object type in a third image of the separate perspective of the scene captured by the third image sensor, wherein the first object type has an expected luminance within the first range of luminance levels; and

identify objects of a second object type in a fourth image of the separate perspective of the scene captured by the fourth image sensor, wherein the second object type has an expected luminance within the second range of luminance levels.

9. The camera system of claim 1 , further comprising:

a third image sensor having the first dynamic range; and

a fourth image sensor having the second dynamic range,

wherein the processor is coupled to the third image sensor and the fourth image sensor, and

wherein the processor is further configured to execute instructions to identify the object based on a third image captured by the third image sensor and a fourth image captured by the fourth image sensor.

10. The camera system of claim 9 ,

wherein the third image sensor has a variable exposure that is adjusted by a controller executing instructions based on luminance, and

wherein the fourth image sensor has a fixed exposure.

11. The camera system of claim 10 , wherein adjusting the variable exposure based on luminance comprises adjusting an exposure duration, an aperture size, or an ISO sensitivity of the third image sensor.

12. The camera system of claim 1 , further comprising:

a first lens optically coupled to the first image sensor and having a first focal length; and

a second lens optically coupled to the second image sensor and having a second focal length,

wherein the first focal length is different than the second focal length, and

wherein, correspondingly, the first image of the scene and the second image of the scene are captured at different focal lengths.

13. The camera system of claim 1 , further comprising:

a first lens optically coupled to the first image sensor and having a first focal length; and

a second lens optically coupled to the second image sensor and having a second focal length,

wherein the first focal length is different than the second focal length,

wherein, correspondingly, the first image of the scene and the second image of the scene are captured at different focal lengths, and

wherein the processor is further configured to execute instructions to determine a distance of the object based on:

the first focal length;

the second focal length;

a focus of the object in the first image of the scene; and

a focus of the object in the second image of the scene.

14. The camera system of claim 1 , wherein the first image and the second image are captured substantially simultaneously.

15. The camera system of claim 1 , wherein the first image sensor and the second image sensor are vertically aligned.

16. The camera system of claim 1 , wherein the camera system is mounted on an autonomous vehicle and used for object detection and avoidance.

17. The camera system of claim 1 , wherein the second image sensor receives light from the scene via a neutral-density filter.

18. The camera system of claim 1 , wherein the first portion of the object exhibiting the first luminance comprises one or more of the first portion of the object illuminating light or the first portion of the object being illuminated by light, and

wherein the second portion of the object exhibiting the second luminance comprises one or more of the second portion of the object illuminating light or the second portion of the object being illuminated by light.

19. A method, comprising:

capturing, by a first image sensor having a first dynamic range corresponding to a first range of luminance levels in a scene, a first image of the scene;

capturing, by a second image sensor having a second dynamic range corresponding to a second range of luminance levels in the scene, a second image of the scene, wherein the second range of luminance levels includes luminance levels that are higher than the first range of luminance levels;

identifying, by a processor coupled to the first image sensor and the second image sensor, a first portion of an object based on the first image of the scene captured by the first image sensor, wherein the first portion of the object exhibits a first luminance corresponding to the first range of luminance levels;

identifying, by the processor, a second portion of the object, different from the first portion, based on the second image of the scene captured by the second image sensor, wherein the second portion of the object exhibits a second luminance corresponding to the second range of luminance levels; and

identifying, by the processor, the object based on the identified first and second portions.

20. A non-transitory, computer-readable medium with instructions stored thereon, wherein the instructions, when executed by a processor, comprise:

receiving, from a first image sensor having a first dynamic range corresponding to a first range of luminance levels in a scene, a first image of the scene, wherein the first image of the scene was captured by the first image sensor;

receiving, from a second image sensor having a second dynamic range corresponding to a second range of luminance levels in the scene, a second image of the scene, wherein the second range of luminance levels includes luminance levels that are higher than the first range of luminance levels;

identifying a first portion of an object based on the first image of the scene captured by the first image sensor, wherein the first portion of the object exhibits a first luminance corresponding to the first range of luminance levels;

identifying a second portion of the object, different from the first portion, based on the second image of the scene captured by the second image sensor, wherein the second portion of the object exhibits a second luminance corresponding to the second range of luminance levels; and

identifying the object based on the identified first and second portions.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 22, 2020
From: WENDEL, ANDREAS; DITTMER, JEREMY; HERMALYN, BRENDAN; INGRAM, BENJAMIN
To: WAYMO LLC
Reel/Frame 053283/0118 →
Continuity (3)
Continuation 16199998 · Nov 26, 2018
Provisional Application 62611206 · Dec 28, 2017
Related Publication 20200351434A1 · Nov 5, 2020
Cited By (1)
US 12,250,447