Image-guided adjustment to super-resolution operations

1. An image processing circuit operative to perform super-resolution (SR) operations, comprising:

a memory to store a plurality of parameter sets of a plurality of artificial intelligence (AI) models that have been trained in an offline training phase;

an image guidance module to detect, using one or more of the AI models, a representative feature in an image sequence received during an online processing phase, the image sequence including a current frame and past frames within a time window;

a parameter decision module, which, during the online processing phase and based on the representative feature, is operative to generate N non-zero weights corresponding to N of the parameter sets, N being an integer greater than one, wherein the representative feature includes at least a scene type classification and an image quality measurement, and wherein in the offline training phase one of the N parameter sets is generated for the scene type classification and another of the N parameter sets is generated for the image quality measurement; and

an SR engine to execute a plurality of layers of AI operations on the current frame to thereby generate a high-resolution image for display, the SR engine operative to:

receive an input feature map at an intermediate layer of the plurality of layers from a previous layer,

perform N AI operations on the input feature map using respective ones of the N parameter sets to generate N outputs, and

compute a weighted sum of the N outputs using respective ones of the N non-zero weights to generate an output feature map to a next layer.

2. The image processing circuit of claim 1 , wherein the parameter decision module is further operative to:

generate weights for two or more pre-trained parameter sets of the AI models based on the representative feature; and

apply the weights to the two or more pre-trained parameter sets to generate adjusted parameters.

3. The image processing circuit of claim 2 , wherein the parameter decision module further comprises an adder operative to:

calculate a weighted sum from the weights and the two or more pre-trained parameter sets to generate the adjusted parameters.

4. The image processing circuit of claim 2 , wherein the parameter decision module further comprises an adder operative to:

compute a weighted sum of kernels of a convolutional neural network (CNN) to obtain a new kernel for the SR engine to process the current frame.

5. The image processing circuit of claim 2 , wherein the representative feature includes an image quality measurement q, and the two or more pre-trained parameter sets are pre-trained with images of image quality measurements q 1 and q 2 , where q 1 <q<q 2 .

6. The image processing circuit of claim 1 , wherein the parameter decision module further includes a multiplexer operative to:

select a parameter set from a plurality of pre-trained parameter sets as a new parameter set based on the measurement of the representative feature; and

output the selected parameter set to the SR engine for processing the current frame.

7. The image processing circuit of claim 1 , further comprising a CNN accelerator operative to perform CNN computations in the processing of the current frame.

8. The image processing circuit of claim 1 , wherein the representative feature includes one or more of: an image quality index, a scene type, a degradation type, a degradation level, a color condition, image resolution, a noise level, a noise type, a video compression parameter, a coding artifacts level, color saturation, sharpness, and contrast.

9. The image processing circuit of claim 1 , wherein the SR engine is further operative to:

remove artifacts in the current frame using the one or more AI models and the parameters adjusted based on a scene type and an image quality index measured from the image sequence.

10. A method for performing super-resolution (SR) operations, comprising:

receiving an image sequence including a current frame and past frames within a time window during an online processing phase;

detecting a representative feature in the image sequence using one or more of a plurality of artificial intelligence (AI) models that have been trained in an offline training phase, wherein the plurality of AI models are defined by a plurality of parameter sets;

generating, during the online processing phase and based on the representative feature, N non-zero weights corresponding to N of the parameter sets, N being an integer greater than one, wherein the representative feature includes at least a scene type classification and an image quality measurement, and wherein in the offline training phase one of the N parameter sets is generated for the scene type classification and another of the N parameter sets is generated for the image quality measurement; and

executing a plurality of layers of AI operations on the current frame to thereby generate a high-resolution image for display, wherein executing the plurality of layers further comprises:

receiving an input feature map at an intermediate layer of the plurality of layers from a previous layer,

performing N AI operations on the input feature map using respective ones of the N parameter sets to generate N outputs, and

computing a weighted sum of the N outputs using respective ones of the N non-zero weights to generate an output feature map to a next layer.

11. The method of claim 10 , further comprising:

generating weights for two or more pre-trained parameter sets of the AI models based on the representative feature; and

applying the weights to the two or more pre-trained parameter sets to generate adjusted parameters.

12. The method of claim 11 , wherein applying the weights further comprises:

calculating a weighted sum from the weights and the two or more pre-trained parameter sets to generate the adjusted parameters.

13. The method of claim 11 , wherein applying the weights further comprises:

computing a weighted sum of kernels of a convolutional neural network (CNN) to obtain a new kernel for processing the current frame.

14. The method of claim 11 , wherein the representative feature includes an image quality measurement q, and the two or more pre-trained parameter sets are pre-trained with images of image quality measurements q 1 and q 2 , where q 1 <q<q 2 .

15. The method of claim 10 , further comprising:

selecting a parameter set from a plurality of pre-trained parameter sets based on the measurement of the representative feature; and

processing the current frame using the one or more AI models with the selected parameter set.

16. The method of claim 10 , wherein processing the current frame further comprises:

performing CNN computations to process the current frame.

17. The method of claim 10 , wherein the representative feature includes one or more of: an image quality index, a scene type, a degradation type, a degradation level, a color condition, image resolution, a noise level, a noise type, a video compression parameter, a coding artifacts level, color saturation, sharpness, and contrast.

18. The method of claim 10 , wherein processing the current frame further comprises:

removing artifacts in the current frame using the one or more AI models and the parameters adjusted based on a scene type and an image quality index measured from the image sequence.