Method and system for tuning a camera image signal processor for computer vision tasks
Image Signal Processing (ISP) optimization framework for computer vision applications is disclosed. The tuning of the ISP is performed automatically and presented as a nonlinear multi-objective optimization problem, followed by solving the problem using an evolutionary stochastic solver. An improved ISP of the embodiments of the invention includes at least features of search space reduction for reducing a number of ISP configurations, remapping the generated population to the reduced search space via mirroring, and global optimization function processing, which allow tuning all the blocks of the ISP at the same time instead of the prior art tuning of each ISP block separately. Also shown that an ISP tuned for image quality performs inferior compared with an ISP trained for a specific downstream image recognition task.
1. A method for tuning an image signal processor (ISP) of a digital camera having configuration parameters, the method comprising:
employing at least one hardware processor for:
(a) capturing a raw image;
(b) processing the raw image by the ISP with the configuration parameters to generate a processed image;
(c) supplying the processed image to a computer vision (CV) system;
(d) measuring a performance of the CV system with regard to a target CV task;
(e) tuning the configuration parameters of the ISP based on the measured performance of the CV system, comprising:
(i) building a multi-objective loss function with regard to the measure of performance; and
(ii) applying an evolutionary algorithm, having a number of trials (population), to the multi-objective loss function to determine the configuration parameters of the ISP as tuned in the step (e), thereby performing a global optimization of at least two or more modules of the ISP simultaneously; and
(f) repeating the steps (b)-(e) using the configuration parameters from the step (e) until a required measure of performance of the CV system is achieved, thereby improving the performance of the CV system for the target CV task.
2. The method of claim 1 , wherein the (ii) step further comprises remapping the population which falls outside of search bounds for the configuration parameters back to the search bounds using a mirroring based remapping technique.
3. The method of claim 1 , wherein the at least two or more modules of the ISP comprise at least two or more of the following:
denoising;
demosaicking;
white balancing;
color correction;
tone mapping;
image sharpening;
JPEG compression.
4. A method for tuning an image signal processor (ISP) of a digital camera having configuration parameters, the method comprising:
employing at least one hardware processor for:
(a) capturing a raw image;
(b) processing the raw image by the ISP with the configuration parameters to generate a processed image;
(c) supplying the processed image to a computer vision (CV) system;
(d) measuring a performance of the CV system with regard to a target CV task;
(e) tuning the configuration parameters of the ISP based on the measured performance of the CV system, comprising reducing a number of combinations of the configuration parameters of the ISP (ISP sets), comprising:
(i) sampling a Latin Hyper-Cube space with regard to the configuration parameters of the ISP;
(ii) measuring the performance of the CV system for the ISP sets sampled in the step (i); and
(iii) selecting ranges for configuration parameters based on those ISP sets which result in the measure of performance of the CV system above a predetermined threshold; and
(f) repeating the steps (b)-(e) using the configuration parameters from the step (e) until a required measure of performance of the CV system is achieved, thereby improving the performance of the CV system for the target CV task.
5. A method for tuning an image signal processor (ISP) of a digital camera having a configuration parameters, the method comprising:
employing at least one hardware processor for:
(a) capturing a raw image;
(b) processing the raw image by the ISP with the configuration parameters to generate a processed image;
(c) supplying the processed image to a computer vision (CV) system;
(d) measuring a performance of the CV system with regard to a target CV task;
(e) tuning the configuration parameters of the ISP based on the measured performance of the CV system, further comprising modifying a search space for the configuration parameters, comprising:
for each configuration parameter:
(i) for a loss for a respective KPI, computing a weight from a p-value of a statistical test chosen so that the p-value is inversely correlated with an impact of said each configuration parameter on the loss;
(ii) scalarizing weighted losses of all KPIs to obtain a single loss;
(iii) determining a modified search range containing those trials which have lowest scalarized loss values; and
determining a modified search space comprising aggregating modified search ranges across all configuration parameters; and
(f) repeating the steps (b)-(e) using the configuration parameters from the step (e) until a required measure of performance of the CV system is achieved, thereby improving the performance of the CV system for the target CV task.
6. A method for tuning an image signal processor (ISP) of a digital camera having configuration parameters, the method comprising:
employing at least one hardward processor for:
(a) capturing a raw image;
(b) processing the raw image by the ISP with the configuration parameters to generate a processed image;
(c) supplying the processed image to a computer vision (CV) system;
(d) measuring a performance of the CV system with regard to a target CV task;
(e) tuning the configuration parameters of the ISP based on the measured performance of the CV system, comprising:
initializing an initial set of estimates of the configuration parameters, comprising:
retrieving an initial set of trials (population);
for at least some of the trials of the initial set of trials, ranking key performance indicator (KPI) values, indicating the performance of the CV system;
determining those configuration parameters which correspond to top ranked KPI values; and
determining the initial set of estimates as said those configuration parameters corresponding to the top ranked KPI values; and
(f) repeating the steps (b)-(e) using the configuration parameters from the step (e) until a required measure of performance of the CV system is achieved, thereby improving the performance of the CV system for the target CV task.
7. The method of claim 6 , wherein the ranking comprises a multi-objective ranking using a scalarization method.
8. The method of claim 7 , wherein the multi-objective ranking comprises using a weighted max-rank method.
9. The method of claim 1 , wherein the tuning comprises tuning at least two or more modules of the ISP at the same time.
10. A method for tuning an image signal processor (ISP) of a digital camera having configuration parameters, the method comprising:
employing at least one harware processor for:
(a) capturing a raw image;
(b) processing the raw image by the ISP with the configuration parameters to generate a processed image;
(c) supplying the processed image to a computer vision (CV) system;
(d) measuring a performance of the CV system with regard to a target CV task;
(e) tuning the configuration parameters of the ISP based on the measured performance of the CV system, further comprising remapping an initial number of trials (population) of configuration parameters which falls outside a search space for the configuration parameters, back onto the search space; and
(f) repeating the steps (b)-(e) using the configuration parameters from the step (e) until a required measure of performance of the CV system is achieved, thereby improving the performance of the CV system for the target CV task.
11. The apparatus of claim 1 , wherein the step (a) comprises one of the following:
capturing the raw image by digital camera;
computer simulating the raw image.