Method for automatic sensor pixel arrangement optimized for multiple camera tasks
A method for optimization of alignment of pixels of X different types on a sensor array for the performance of N different camera tasks, where X and N are integers greater than 1, includes obtaining an output of each of the X types of pixel on the sensor array for an input image; evaluating the quality of the output for each of the N different camera tasks; and obtaining an optimal pixel alignment pattern by adjustment of a potential pixel alignment pattern with respect to performance of the N different camera tasks subject to a user-determined balance of priorities among the N different camera tasks.
1. A method for optimization of alignment of pixels of X different types on a sensor array for a camera capable of performing N different camera tasks, where X and N are integers greater than 1; the method comprising:
obtaining an output of each of the X types of pixel on the sensor array for an input image;
evaluating a quality of the output for each of the N different camera tasks; and
obtaining an optimal pixel alignment pattern by adjustment of a potential pixel alignment pattern with respect to performance of the N different camera tasks subject to a user-determined balance of priorities among the N different camera tasks, wherein obtaining the optimal pixel alignment pattern comprises evaluating all possible combinations of pixel alignments on the sensor array.
2. A method for optimization of alignment of pixels of X different types on a sensor array for a camera capable of performing N different camera tasks, where X and N are integers greater than 1; the method comprising:
obtaining an output of each of the X types of pixel on the sensor array for an input image;
evaluating a quality of the output for each of the N different camera tasks; and
obtaining an optimal pixel alignment pattern by adjustment of a potential pixel alignment pattern with respect to performance of the N different camera tasks subject to a user-determined balance of priorities among the N different camera tasks, wherein obtaining the optimal pixel alignment pattern comprises using one or more rule-based algorithms.
3. A method for optimization of alignment of pixels of X different types on a sensor array for a camera capable of performing N different camera tasks, where X and N are integers greater than 1; the method comprising:
obtaining an output of each of the X types of pixel on the sensor array for an input image;
evaluating a quality of the output for each of the N different camera tasks; and
obtaining an optimal pixel alignment pattern by adjustment of a potential pixel alignment pattern with respect to performance of the N different camera tasks subject to a user-determined balance of priorities among the N different camera tasks, wherein obtaining the optimal pixel alignment pattern comprises using machine learning.
4. The method of claim 3 , wherein using machine learning comprises utilizing a trainable pixel alignment layer.
5. The method of claim 4 , wherein utilizing a trainable pixel alignment layer comprises using a continuous relaxation technique.
6. The method of claim 3 , wherein machine learning comprises utilizing a reinforcement learning technique.
7. A method for optimization of alignment of pixels of X different types on a sensor array for a camera capable of performing N different camera tasks, where X and N are integers greater than 1; the method comprising:
obtaining an output of each of the X types of pixel on the sensor array for an input image;
evaluating a quality of the output for each of the N different camera tasks; and
obtaining an optimal pixel alignment pattern by adjustment of a potential pixel alignment pattern with respect to performance of the N different camera tasks subject to a user-determined balance of priorities among the N different camera tasks,
wherein the sensor array comprises a tessellation of identical pixel blocks, each block comprising an arrangement of pixels of the X different types; and
wherein obtaining the optimal pixel alignment pattern comprises obtaining an optimal arrangement of pixels of the X different types in one block.
8. The method of claim 7 , wherein obtaining the optimal arrangement of pixels in each block comprises evaluating all possible combinations of pixel alignments in the block.
9. The method of claim 7 , wherein obtaining the optimal pixel alignment pattern comprises using one or more rule-based algorithms.
10. The method of claim 7 , wherein obtaining the optimal pixel alignment pattern comprises using machine learning.
11. The method of claim 10 , wherein using machine learning comprises utilizing a trainable pixel alignment layer.
12. The method of claim 11 , wherein utilizing a trainable pixel alignment layer comprises using a continuous relaxation technique.
13. The method of claim 10 , wherein using machine learning comprises utilizing a reinforcement learning technique.
14. A method for optimization of alignment of pixels of X different types on a sensor array for a camera capable of performing N different camera tasks, where X and N are integers greater than 1; the method comprising:
obtaining an output of each of the X types of pixel on the sensor array for an input image;
evaluating a quality of the output for each of the N different camera tasks; and
obtaining an optimal pixel alignment pattern by adjustment of a potential pixel alignment pattern with respect to performance of the N different camera tasks subject to a user-determined balance of priorities among the N different camera tasks,
wherein the sensor array is a tessellation of first and second pixel blocks, each of the first blocks comprising a first arrangement of pixels of the X different types, and each of the second blocks comprising a second arrangement, different from the first arrangement of pixels of the X different types; and
wherein obtaining the optimal pixel alignment pattern comprises obtaining an optimal arrangement of pixels in a first block and in a second block.
15. A method for optimization of alignment of pixels of X different types on an image sensor array for for a camera capable of performing N different camera tasks, where X and N are integers greater than 1; the method comprising:
simulating pixel response for training images, such that for each training image, X sensor images are output, each sensor image corresponding uniquely to pixels of only one of the X different types, positioned at each location possible on the image sensor array;
generating, in a trainable pixel alignment layer, a pixel alignment pattern according to which pixels of the X different types are distributed over the image sensor array locations as a 2D tessellation;
subsampling each of the X sensor images according to the pixel alignment pattern to output X corresponding subsampled sensor images; and
simultaneously training the pixel alignment layer and a stack of N neural networks, each neural network corresponding uniquely to only one of the N different camera tasks and accepting and processing one or more of the X subsampled sensor images, making discrete, categorical choices of pixels of the X different types such that the pixel alignment pattern is optimized with respect to performance of the N different camera tasks subject to a user-determined balance of priorities among the N different camera tasks.
16. The method of claim 15 , wherein simultaneously training the pixel alignment layer and N neural networks comprises concurrent use of an inner feedback loop, feeding a combined performance value output derived from the N outputs from the stack back to each of the N neural networks, and an outer feedback loop, feeding the combined performance value back to the trainable pixel alignment layer.
17. The method of claim 16 , wherein the combined performance value output from the stack comprises a total loss value comprising a combination of performance value outputs from each of the N neural networks, each performance value output being a function of loss value for the corresponding neural network and of at least one user-supplied weighting parameter for the camera task corresponding to that neural network.
18. The method of claim 17 , wherein the function of loss value and at least one user-supplied weighting parameter for one of the neural networks comprises a linear function of that loss value.
19. The method of claim 15 , wherein at least one of the N tasks depends primarily on image quality captured by pixels at the sensor array and wherein at least another one of the N tasks does not depend primarily on image quality captured by pixels at the sensor array.
20. A system for optimization of alignment of pixels of X different types on an image sensor array for a camera capable of performing N different camera tasks, where X and N are integers greater than 1; the system comprising:
a pixel response simulator;
a trainable pixel alignment layer;
a subsampler; and
a stack of N trainable neural networks, each neural network corresponding uniquely to only one of the N different camera tasks;
wherein the pixel response simulator is configured to operate on training images to deliver to the subsampler, for each training image, X sensor images, each sensor image corresponding uniquely to pixels of only one type, positioned at each location possible on the image sensor array;
wherein the trainable pixel alignment layer is configured to generate and deliver to the subsampler a pixel alignment pattern according to which pixels of the X different types are distributed over the image sensor array locations as a 2D tessellation;
wherein the subsampler is configured to subsample each of the X sensor images according to the pixel alignment pattern and to deliver one or more subsampled output images to one or more of the neural networks to be processed therein; and
wherein the N neural networks and the pixel alignment layer are simultaneously trained using inner feedback loops and an outer feedback loop respectively, making discrete, categorical choices of pixels of the X different types such that the pixel alignment pattern is optimized with respect to performance of the N camera tasks subject to a user-determined balance of priorities among the N camera tasks.