Application processor including neural processing unit and operating method thereof
An application processor includes a neural processing unit configured to convert an input image into a first image based on a first pattern and generate a second image using a neural network, the second image compensating for the conversion; and an image signal processor including a plurality of pipelines configured to perform image signal processing, the plurality of pipelines including at least a first pipeline configured to receive the first image and a second pipeline configured to receive the second image.
1. An application processor comprising:
a neural processing unit configured to convert an input image into a first image based on a first pattern and generate a second image using a neural network, the second image compensating for the conversion; and
an image signal processor including a plurality of pipelines configured to perform image signal processing, the plurality of pipelines including at least a first pipeline configured to receive the first image and a second pipeline configured to receive the second image,
wherein the second pipeline is connected to a back end of the first pipeline such that the second pipeline is configured to receive an output of the first pipeline, and
wherein the second pipeline is configured to compensate for data lost about the input image due to converting the input image into the first image, and the compensating is based on the second image.
2. The application processor of claim 1 , wherein
the first pattern includes a Bayer pattern, and
the input image has a second pattern different from the first pattern.
3. The application processor of claim 1 , wherein the second image represents luminance information of the input image.
4. The application processor of claim 1 , wherein
the first pipeline is configured to output an image by performing the image signal processing on the first image; and
the second pipeline is configured to generate a third image, of a red, green, blue (RGB) type, by performing color interpolation on the image output from the first pipeline.
5. The application processor of claim 4 , wherein the second pipeline is further configured to apply filtering to the third image based on the second image.
6. The application processor of claim 4 , wherein the first pipeline is configured to perform at least one of removing noise from the first image or detecting a bad pixel in the first image.
7. The application processor of claim 1 , wherein the neural network includes a sub network configured to learn a feature map based on the input image and a predicted image, and to output an image corresponding to the feature map.
8. The application processor of claim 7 , wherein
the feature map is based on a loss function including a difference value between an original image in the first pattern and the predicted image, and
the sub network is further configured to be trained based on the loss function and to output the first image from the input image.
9. The application processor of claim 7 , wherein
the feature map is based on a loss function including a difference value between an original image representing luminance information and the predicted image, and
the sub network is further configured to be trained based on the loss function and to output the second image from the input image.
10. The application processor of claim 7 , wherein
the feature map is based on a loss function including a difference value between a noise removed original image and the predicted image, and
the sub network is further configured to be trained based on the loss function and to output a noise-removed image from the input image.
11. The application processor of claim 1 , further comprising:
a memory device configured to receive at least one of the first image or the second image from the neural processing unit and to transmit the at least one of the first image and the second image to the image signal processor.
12. An image processing system comprising:
an image sensor including a plurality of pixels divided into groups, each of the groups including four adjacent pixels corresponding to a same color; and
the application processor of claim 1 ,
wherein the input image is provided to the neural processing unit from the image sensor.
13. An application processor comprising:
a neural processing unit configured to generate a first image from an input video using a neural network, the input video being provided from an image sensor and including a plurality of frames, and the first image representing information regarding the plurality of frames; and
an image signal processor configured to convert at least one of the plurality of frames into a second image based a Bayer pattern and to perform image signal processing on the second image,
wherein the image signal processing includes compensating for data lost about the input image due to converting the at least one of the plurality of frames into the second image, and the compensating is based on the first image.
14. The application processor of claim 13 , wherein the information regarding the plurality of frames represents luminance information of the plurality of frames.
15. The application processor of claim 13 , wherein the image sensor includes a plurality of pixels divided into groups having four adjacent pixels corresponding to a same color.
16. The application processor of claim 15 , wherein the second image is generated by summing image signals respectively generated by the four adjacent pixels included in the plurality of frames.
17. The application processor of claim 13 , wherein the image signal processor is further configured to perform color interpolation on adjacent pixels in the second image such that a third image of a red, green, blue (RGB) type is generated and to apply filtering to the third image based on the first image.
18. An operating method of an application processor, the operating method comprising:
receiving an input image;
converting the input image into a first image based on a Bayer pattern using a neural network;
generating a second image using the neural network, the second image compensating for the conversion; and
performing image signal processing on the first image,
wherein performing the image signal processing includes compensating for data lost about the input image due to converting the input image into the first image, and the compensating is based on the second image.
19. The operating method of claim 18 , wherein
the second image represents luminance information regarding the input image, and
the performing of the image signal processing on the first image includes performing color interpolation on adjacent pixels in the first image such that a third image of a red, green, blue (RGB) type is generated and applying filtering to the third image based on the second image.
20. The operating method of claim 18 , further comprising:
removing noise from the input image using the neural network before the converting of the input image.