METHODS AND APPARATUS FOR COMPUTER VISION BASED ON MULTI-STREAM FEATURE-DOMAIN FUSION
A computer-vision pipeline is organized as a closed loop of a sensor-processing phase, an image-processing phase, and an object-detection phase, each comprising a respective phase processor coupled to a master processor. The sensor-processing phase creates multiple exposure images, and derives multi-exposure multi-scale zonal illumination-distributions, to be processed independently in the image-processing phase. In a first implementation of the object-detection phase, extracted exposure-specific features are pooled prior to overall object detection. In a second implementation, exposure-specific objects, detected from the exposure-specific features, are fused to produce the sought objects of a scene under consideration. The two implementations enable detecting fine details of a scene under diverse illumination conditions. The master processor performs loss-function computations to derive updated training parameters of the processing phases. Several experiments applying a core method of operating the computer-vision pipelines, and variations thereof, ascertain performance gain under challenging illumination conditions.
1 . A method of detecting objects from camera-produced images comprising:
generating multiple raw exposure-specific images for a scene;
performing for said multiple raw exposure-specific images respective processes of image enhancement to produce respective processed exposure-specific images;
extracting from said processed exposure-specific images respective sets of exposure-specific features collectively constituting a superset of features;
fusing constituent exposure-specific sets of features of said superset of features to form a set of fused features;
identifying a set of candidate objects from said set of fused features; and
pruning said set of candidate objects to produce a set of objects within said scene.
2 . A method of detecting objects from camera-produced images comprising:
generating multiple raw exposure-specific images for a scene;
performing for said multiple raw exposure-specific images respective processes of image enhancement to produce respective processed exposure-specific images;
extracting from said processed exposure-specific images respective sets of exposure-specific features collectively constituting a superset of features;
identifying, using said respective sets of exposure-specific features, exposure-specific sets of candidate objects;
fusing said exposure-specific sets of candidate objects to form a fused set of candidate objects; and
pruning said set of candidate objects to produce a set of objects within said scene.
3 . The method of claim 2 further comprising deriving for each raw exposure-specific image a respective multi-level regional illumination distribution for use in computing respective exposure settings.
4 . A method of detecting objects from camera-produced images comprising:
generating multiple raw exposure-specific images for a scene;
deriving for each raw exposure-specific image a respective multi-level regional illumination distribution for use in computing respective exposure settings;
performing for said multiple raw exposure-specific images respective processes of image enhancement to produce respective processed exposure-specific images;
extracting from said processed exposure-specific images respective sets of exposure-specific features collectively constituting a superset of features;
recognizing a set of candidate objects using said superset of features; and
pruning said set of candidate objects to produce a set of objects within said scene.
5 . The method of claim 4 further comprising selecting image regions, for use in said deriving, categorized in a predefined number of levels so that each region of a level, other than a last level of said predefined number of levels, encompasses an integer number of regions of each subsequent level.
6 . The method of claim 4 wherein said respective processes of image enhancement are performed according to one of:
sequentially using a single image-signal-processor;
using multiple pipelined image signal processors operating cooperatively and concurrently; or
using multiple pipelined image signal processors operating independently and concurrently.
7 . The method of claim 4 wherein said recognizing comprises:
fusing constituent exposure-specific sets of features of said superset of features to form a set of fused features; and
identifying a set of candidate objects from said set of fused features.
8 . The method of claim 4 wherein said recognizing comprises:
identifying, using said respective sets of exposure-specific features, exposure-specific sets of candidate objects; and
fusing said exposure-specific sets of candidate objects to form a fused set of candidate objects.
9 . The method of claim 8 further comprising:
determining objectness of each detected object of said fused set of candidate objects; and
pruning said fused set of candidate objects according to a non-maximum-suppression criterion.
10 . The method of claim 8 further comprising:
determining objectness of each detected object of said superset of detected objects; and
pruning said fused superset of detected objects according to a keep-best-loss principle.
11 . The method of claim 4 wherein said respective processes of image enhancement for each exposure-specific image comprise:
raw image contrast stretching, using lower and upper percentiles for pixel-wise affine mapping;
image demosaicing;
image resizing;
a pixel-wise power transformation; and
pixel-wise affine transformation with learned parameters.
12 . The method of claim 4 further comprising:
updating parameters pertinent to said generating, deriving, performing, extracting, and recognizing to produce respective updated parameters; and
disseminating said respective updated parameters to relevant hardware processors per-forming said generating, deriving, performing, extracting, and recognizing.
13 . The method of claim 12 wherein said updating comprises processes of:
establishing a loss function; and
pruning backpropagation loss components.
14 . The method of claim 12 wherein said disseminating comprises employing a network of hard-ware processors coupled to a plurality of memory devices storing processor-executable instructions for performing said generating, deriving, performing, extraction, and recognizing.
15 . An apparatus for detecting objects, from camera-produced images of a time-varying scene, comprising:
a hardware master processor coupled to a pool of hardware intermediate processors;
a sensing-processing device comprising:
a sensor;
a sensor-control device comprising a neural auto-exposure controller, coupled to a light-collection component, configured to:
generate a specified number of time-multiplexed exposure-specific raw SDR images; and
derive for each exposure-specific raw SDR image respective multi-level luminance histograms;
an image-processing device configured to perform predefined image-enhancing procedures for each said raw SDR image to yield multiple exposure-specific processed images;
a features-extraction device configured to extract from said multiple exposure-specific processed images respective sets of exposure-specific features collectively constituting a superset of features;
an objects-detection device configured to identify a set of candidate objects using said superset of features; and
a pruning module configured to filter said set of candidate objects to produce a set of pruned objects within said time-varying scene.
16 . The apparatus of claim 15 wherein:
said hardware master-processor is communicatively coupled to each hardware intermediate processor through one of:
a dedicated path;
a shared bus; or
a switched path.
17 . The apparatus of claim 16 wherein each of said sensing-processing device, image-processing device, features-extraction device, and objects-detection device is coupled to a respective hard-ware intermediate processor of said pool of hardware intermediate processors, thereby facilitating dissemination of control data through the apparatus.
18 . The apparatus of claim 15 further comprising an illumination-characterization module, for deriving said respective multi-level luminance histograms, configured to select image-illumination regions for each level of a predefined number of levels, so that each region of a level, other than a last level of said predefined number of levels, encompasses an integer number of regions of each subsequent level.
19 . The apparatus of claim 15 wherein said image-processing device is configured as one of:
a single image-signal-processor (ISP) sequentially performing said predefined image enhancing procedures for said specified number of time-multiplexed exposure-specific raw SDR images;
a plurality of pipelined image-processing units operating cooperatively and concurrently to execute said image-enhancing procedure; or
a plurality of image-signal-processors, operating independently and concurrently, each processing a respective raw SDR image.
20 . The apparatus of claim 15 wherein said objects-detection device comprises:
a features-fusing module configured to fuse said respective sets of exposure-specific features of said superset of features to form a set of fused features; and
a detection module configured to identify a set of candidate objects from said set of fused features.
21 . The apparatus of claim 15 wherein said objects-detection device comprises:
a plurality of detection modules, each configured to identify, using said respective sets of exposure-specific features, exposure-specific sets of candidate objects; and
an objects-fusing module configured to fuse said exposure-specific sets of candidate objects to form a fused set of candidate objects.
22 . The apparatus of claim 15 further comprising a control module configured to cause said master processor to:
derive, based on said set of pruned objects, updated parameters pertinent to said:
sensing-processing device;
image-processing device;
features-extraction device; and
objects-detection device; and
disseminate said updated parameters through said pool of hardware processors.
23 . The apparatus of claim 22 wherein said control module is configured to determine derivatives of a loss function, based on said pruned set of objects, to produce said updated device parameters.
24 . The apparatus of claim 23 further comprising a module for selecting downstream control data according to one of:
a method based on keeping best loss, or
a method based on non-maximal suppression.
25 . The apparatus of claim 15 further comprising a module for tracking, for determining a lower bound of a capturing time interval, processing durations within each of:
the sensing-processing device;
the image-processing device;
the features-extraction device; and
the objects-detection device.