Granted Patent B2

US 12,482,068 · App. 17/722,261 · Granted Nov 25, 2025

Inventors: Emmanuel Luc Julien Onzon (Munich, DE); Felix Heide (Palo Alto, CA); Fahim Mannan (Montreal, CA)

Assignee: TORC CND ROBOTICS, INC.

G06T5/50G06F18/241G06F18/24133G06N3/045G06N3/084G06N20/00G06T5/60G06T5/73G06V10/454G06T2207/20081G06T2207/20084G06T2207/20182

Abstract

Claims (60)

1 . A computer-implemented method for determining an auto-exposure value of a low dynamic range (LDR) sensor for use in high dynamic range (HDR) object detection, the method comprising:

employing at least one hardware processor for:

forming an auto-exposure neural network for predicting exposure values for the LDR sensor driven by a downstream object detection neural network in real time;

training the auto-exposure neural network jointly, end-to-end together with (i) the object detection neural network and (ii) an image signal processing (ISP) pipeline, thereby yielding a trained auto-exposure neural network; and

using the trained auto-exposure neural network to generate an optimal exposure value for the LDR sensor and the downstream object detection neural network for the HDR object detection.

2 . The computer-implemented method of claim 1 , wherein the forming comprises forming a global image feature neural network.

3 . The computer-implemented method of claim 1 , wherein the forming comprises forming a semantic feature neural network.

4 . The computer-implemented method of claim 1 , wherein the forming comprises forming a Hybrid neural network, comprising both a global image feature neural network and a semantic feature neural network.

5 . The computer-implemented method of claim 1 , further comprising, prior to the training, forming a training dataset of images, comprising:

capturing a set of HDR images by an HDR sensor in a real life environment; and

for each HDR image from the set of HDR images, forming a corresponding linear HDR image, thereby forming the training dataset.

6 . The computer-implemented method of claim 1 , further comprising, prior to the training, forming a training dataset of images, comprising:

by an HDR sensor, for each HDR image captured in a real life environment, outputting “n” linear LDR images with different exposures selected so that a combined dynamic range of the “n” linear LDR images covers a dynamic range of each HDR image.

7 . The computer-implemented method of claim 6 , wherein the forming the training dataset further comprises fusing the “n” linear LDR images into a corresponding linear HDR image Indr.

8 . The computer-implemented method of claim 7 , wherein the fusing further comprises fusing taking into account a weighted average of pixel values across “n” LDR images with a weight equal to the inverse of an associated noise variance.

9 . The computer-implemented method of claim 5 , wherein the training further comprises simulating a simulated raw LDR image from each corresponding linear HDR image, and using the simulated raw LDR image for the training of the auto-exposure neural network.

10 . The computer-implemented method of claim 9 , wherein:

the training further comprises a plurality of training operations, and per each training operation of the plurality of training operations, the training comprises:

simulating first and second simulated raw LDR images derived from respective first and second linear HDR images and corresponding to two consecutive or closely following frames;

training the first simulated raw LDR image with a random exposure shift; and

training the second simulated raw LDR image with an exposure value predicted by the auto-exposure neural network based on the training of the first simulated raw LDR image.

11 . The computer-implemented method of claim 9 wherein the simulating comprises scaling and quantization of each corresponding linear HDR image.

12 . The computer-implemented method of claim 9 , wherein the simulating further comprises simulating a radiant power per pixel Psim for each simulated raw LDR image as a Bayer pattern sampling of the linear HDR image Indr.

13 . The computer-implemented method of claim 9 , wherein the simulating further comprises adding noise to each simulated raw LDR image to mimic a noise distribution of the LDR sensor.

14 . The computer-implemented method of claim 2 , wherein the forming the global image feature neural network comprises generating histograms from a raw LDR image captured by the LDR sensor at a number of different scales, including a coarse histogram for an entire raw LDR image, and respective finer histograms for corresponding smaller sections of the raw LDR image.

15 . The computer-implemented method of claim 14 , wherein the generating of the histograms comprises generating histograms from one of the following:

green pixel values of the raw LDR image;

luminance pixel values of the raw LDR image;

red pixel values of the raw LDR image; and

blue pixel values of the raw LDR image.

16 . The computer-implemented method of claim 14 , further comprising performing one-dimensional convolution operations of the histograms, followed by dense layer operations on the results of convolution operations.

17 . The computer-implemented method of claim 3 , wherein the forming the semantic feature neural network further comprises:

using an output from a feature extractor ResNet from the object detection neural network as an input to the semantic feature neural network, followed by channel compression to produce a compressed feature map (CFM);

performing pyramid pooling of the CFM at different scales; and

concatenating and densely connecting the results of the pooling.

18 . The computer-implemented method of claim 4 , wherein the training is performed in one of the following modes:

a first mode, comprising:

training the semantic feature neural network alone;

next, adding the global image feature neural network; and

repeating training of both the global image feature neural network and the semantic feature neural network together, following the same training procedure;

or

a second mode, comprising:

training both the global image feature neural network and the semantic feature neural network jointly together.

19 . The computer-implemented method of claim 1 , wherein the using further comprises one or more of the following:

predicting an optimal exposure value for a next frame; and

aggregating predicted exposure values across a number of consecutive frames.

20 . A system for determining an auto-exposure value of a low dynamic range (LDR) sensor for use in high dynamic range (HDR) object detection, the system comprising:

a processor, and a memory having computer executable instructions stored thereon for execution by the processor, causing the processor to:

form an auto-exposure neural network for predicting exposure values for the LDR sensor driven by a downstream object detection neural network in real time;

train the auto-exposure neural network jointly, end-to-end together with (i) the object detection neural network and (ii) an image signal processing (ISP) pipeline, thereby yielding a trained auto-exposure neural network; and

use the trained auto-exposure neural network to generate an optimal exposure value for the LDR sensor and the downstream object detection neural network for the HDR object detection.

21 . A computer vision system comprising:

a low dynamic range sensor (LDR) for use in high dynamic range (HDR) object detection;

an image signal processor (ISP) for processing a raw LDR image from the LDR sensor and outputting a processed image; and

an object detection neural network for further processing the processed image from the ISP;

the computer vision system further comprising an apparatus for determining an auto-exposure value of the LDR sensor, the apparatus comprising:

a processor, and a memory having computer executable instructions stored thereon for execution by the processor, causing the processor to:

form an auto-exposure neural network for predicting exposure values for the LDR sensor driven by the object detection neural network in real time;

train the auto-exposure neural network jointly, end-to-end together with (i) the object detection neural network and (ii) the ISP, thereby yielding a trained auto-exposure neural network; and

use the trained auto-exposure neural network to generate an optimal exposure value for the LDR sensor.

Assignments (2)

Continuity (6)

References Cited (173)