Image signal processor for processing images
Techniques are provided for using one or more machine learning systems to process input data including image data. The input data including the image data can be obtained, and at least one machine learning system can be applied to at least a portion of the image data to determine at least one color component value for one or more pixels of at least the portion of the image data. Based on application of the at least one machine learning system to at least the portion of the image data, output image data for a frame of output image data can be generated. The output image data includes at least one color component value for one or more pixels of the frame of output image data. Application of the at least one machine learning system causes the output image data to have a reduced dimensionality relative to the input data.
1. A method of processing image data using one or more machine learning systems, the method comprising:
obtaining input data including image data;
applying at least one machine learning system to the image data to determine at least one color component value for one or more pixels of the image data; and
generating, based on application of the at least one machine learning system to the image data, output image data for a frame of output image data, the output image data including at least one color component value for one or more pixels of the frame of output image data, wherein application of the at least one machine learning system to the input data causes the output image data to have a reduced dimensionality relative to the input data.
2. The method of claim 1 , wherein the image data includes raw image data.
3. The method of claim 1 , wherein the image data includes a patch of pixels from a frame of image data, and wherein the at least one machine learning system is applied to the patch of pixels.
4. The method of claim 1 , wherein the image data includes a patch of raw image data, the patch of raw image data including a subset of pixels of a frame of raw image data.
5. The method of claim 4 , wherein the frame of raw image data includes raw image data from one or more image sensors filtered by a color filter array.
6. The method of claim 1 , wherein the at least one machine learning system includes at least one neural network.
7. The method of claim 1 , wherein applying the at least one machine learning system to the image data includes:
applying one or more strided convolutional filters to the image data to generate reduced resolution data representative of the image data, each strided convolutional filter of the one or more strided convolutional filters including an array of weights.
8. The method of claim 1 , wherein applying the at least one machine learning system to the image data results in a plurality of color component values being determined for each pixel of the one or more pixels of the frame of output image data.
9. The method of claim 1 , wherein output image data includes a patch of output image data, the patch of output image data including a subset of pixels of the frame of output image data.
10. The method of claim 1 , further comprising:
obtaining additional data for augmenting the obtained image data, the additional data including at least one or more of tone data, radial distance data, or auto white balance (AWB) gain data.
11. The method of claim 1 , wherein the image data is captured by an image sensor.
12. The method of claim 1 , wherein the image data is captured by a plurality of image sensors.
13. An apparatus for processing image data using one or more machine learning systems, comprising:
a memory configured to store input data including image data; and
a processor coupled to the memory and configured to:
obtain the input data including the image data;
apply at least one machine learning system to the image data to determine at least one color component value for one or more pixels of the image data; and
generate, based on application of the at least one machine learning system to the image data, output image data for a frame of output image data, the output image data including at least one color component value for one or more pixels of the frame of output image data, wherein application of the at least one machine learning system to the input data causes the output image data to have a reduced dimensionality relative to the input data.
14. The apparatus of claim 13 , wherein the image data includes raw image data.
15. The apparatus of claim 13 , wherein the image data includes a patch of pixels from a frame of image data, and wherein the at least one machine learning system is applied to the patch of pixels.
16. The apparatus of claim 13 , wherein the image data includes a patch of raw image data, the patch of raw image data including a subset of pixels of a frame of raw image data.
17. The apparatus of claim 16 , wherein the frame of raw image data includes raw image data from one or more image sensors filtered by a color filter array.
18. The apparatus of claim 13 , wherein the at least one machine learning system includes at least one neural network.
19. The apparatus of claim 13 , wherein applying the at least one machine learning system to the image data includes:
applying one or more strided convolutional filters to the image data to generate reduced resolution data representative of the image data, each strided convolutional filter of the one or more strided convolutional filters including an array of weights.
20. The apparatus of claim 13 , wherein applying the at least one machine learning system to the image data results in a plurality of color component values being determined for each pixel of the one or more pixels of the frame of output image data.
21. The apparatus of claim 13 , wherein output image data includes a patch of output image data, the patch of output image data including a subset of pixels of the frame of output image data.
22. The apparatus of claim 13 , wherein the processor is configured to:
obtain additional data for augmenting the obtained image data, the additional data including at least one or more of tone data, radial distance data, or auto white balance (AWB) gain data.
23. The apparatus of claim 13 , wherein the image data is captured by an image sensor.
24. The apparatus of claim 13 , wherein the image data is captured by a plurality of image sensors.
25. The apparatus of claim 13 , further comprising a camera for capturing pictures.
26. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to:
obtain input data including image data;
apply at least one machine learning system to the image data to determine at least one color component value for one or more pixels of the image data; and
generate, based on application of the at least one machine learning system to the image data, output image data for a frame of output image data, the output image data including at least one color component value for one or more pixels of the frame of output image data, wherein application of the at least one machine learning system to the input data causes the output image data to have a reduced dimensionality relative to the input data.
27. The non-transitory computer-readable medium of claim 26 , wherein the image data includes raw image data.
28. The non-transitory computer-readable medium of claim 26 , wherein the image data includes a patch of pixels from a frame of image data, and wherein the at least one machine learning system is applied to the patch of pixels.
29. The non-transitory computer-readable medium of claim 26 , wherein the image data includes a patch of raw image data, the patch of raw image data including a subset of pixels of a frame of raw image data.
30. The non-transitory computer-readable medium of claim 29 , wherein the frame of raw image data includes raw image data from one or more image sensors filtered by a color filter array.