Method and system for tuning a camera image signal processor for computer vision tasks
Image Signal Processing (ISP) optimization framework for computer vision applications is disclosed. The tuning of the ISP is performed automatically and presented as a nonlinear multi-objective optimization problem, followed by solving the problem using an evolutionary stochastic solver. An improved ISP of the embodiments of the invention includes at least features of search space reduction for reducing a number of ISP configurations, remapping the generated population to the reduced search space via mirroring, and global optimization function processing, which allow tuning all the blocks of the ISP at the same time instead of the prior art tuning of each ISP block separately. Also shown that an ISP tuned for image quality performs inferior compared with an ISP trained for a specific downstream image recognition task.
1. An apparatus for processing image data, comprising:
at least one memory; and
at least one processor coupled to the at least one memory and configured to:
process image data to generate a first output image using a first set of configuration parameters for a computer vision operation, wherein the first set of configuration parameters are based on a first function associated with a performance of one or more computer vision systems;
output the first output image to a computer vision system for performing the computer vision operation;
process the image data to generate a second output image using a second set of configuration parameters, wherein the second set of configuration parameters are based on a second function associated with image quality, and wherein the second function is different than the first function; and
output the second output image data for visual output.
2. The apparatus of claim 1 , wherein the at least one processor is an image signal processor (ISP).
3. The apparatus of claim 1 , wherein the image data is raw image data from an image sensor.
4. The apparatus of claim 1 , wherein the computer vision operation comprises at least one of an object detection operation, a segmentation operation, a keypoint detection operation, or an image classification operation.
5. The apparatus of claim 1 , wherein the at least one processor is tuned for multiple computer vision operations based at least on the first set of configuration parameters.
6. The apparatus of claim 1 , wherein the first set of configuration parameters are based on an evolutionary algorithm applied to the first function associated with the performance of the computer vision system, and wherein the second set of configuration parameters are based on the evolutionary algorithm applied to the second function associated with the image quality.
7. The apparatus of claim 6 , wherein the first function associated with the performance of the computer vision system is a first multi-objective loss function, and wherein the second function associated with the image quality is a second multi-objective loss function.
8. A method for processing image data, comprising:
processing, using at least one processor, image data to generate a first output image using a first set of configuration parameters for a computer vision operation, wherein the first set of configuration parameters are based on a first function associated with a performance of one or more computer vision systems;
outputting the first output image to a computer vision system for performing the computer vision operation;
processing, using the at least one processor, the image data to generate a second output image using a second set of configuration parameters, wherein the second set of configuration parameters are based on a second function associated with image quality, wherein the second function is different than the first function; and
outputting the second output image data for visual output.
9. The method of claim 8 , wherein the at least one processor is an image signal processor (ISP).
10. The method of claim 8 , wherein the image data is raw image data from an image sensor.
11. The method of claim 8 , wherein the computer vision operation comprises at least one of an object detection operation, a segmentation operation, a keypoint detection operation, or an image classification operation.
12. The method of claim 8 , wherein the at least one processor is tuned for multiple computer vision operations based at least on the first set of configuration parameters.
13. The method of claim 8 , wherein the first set of configuration parameters are based on an evolutionary algorithm applied to the first function associated with the performance of the computer vision system, and wherein the second set of configuration parameters are based on the evolutionary algorithm applied to the second function associated with the image quality.
14. The method of claim 13 , wherein the first function associated with the performance of the computer vision system is a first multi-objective loss function, and wherein the second function associated with the image quality is a second multi-objective loss function.
15. The apparatus of claim 2 , wherein the ISP is tuned using the first set of configuration parameters to perform the computer vision operation.
16. The apparatus of claim 15 , wherein the ISP is tuned using the second set of configuration parameters to generate output images for visual output.
17. The method of claim 9 , wherein the ISP is tuned using the first set of configuration parameters to perform the computer vision operation.
18. The method of claim 17 , wherein the ISP is tuned using the second set of configuration parameters to generate output images for visual output.
19. The apparatus of claim 1 , wherein the image data includes a first frame of image data processed to generate the first output image and a second frame of the image data processed to generate the second output image.
20. The method of claim 8 , wherein the image data includes a first frame of image data processed to generate the first output image and a second frame of the image data processed to generate the second output image.