Method for code-level super resolution and method for training super resolution model therefor
The present disclosure provides a method for code-level super-resolution imaging and a method of training a super-resolution model therefor. According to some embodiments, the present disclosure provides a method of training a super-resolution model based on the encoded codes of an inputted picture and a method for code-level super-resolution imaging by using a pre-trained super-resolution model.
1 . A method of training to perform super-resolution imaging at a code level, the method comprising:
receiving an input picture;
generating a lowered-resolution picture from the received input picture;
generating input encoded codes by encoding the lowered-resolution picture;
providing the input encoded codes into a super-resolution model to generate, as prediction encoded codes, encoded codes of a high-resolution picture corresponding to the input encoded codes; and
training the super-resolution model based on a loss function defined as a combination of a first difference and a second difference, the first difference representing a difference between the input picture and a decoded version of the prediction encoded codes, and the second difference representing a difference between encoded codes of the input picture and the prediction encoded codes.
2 . The method of claim 1 , wherein the generation of the input encoded codes comprises down-sampling the input picture.
3 . The method of claim 1 , wherein the training of the super-resolution model comprises:
determining a perceptual loss by comparing a feature map extracted from a layer of the super-resolution model and a correct-answer set.
4 . A non-transitory computer-readable medium storing instructions thereon, the instructions when executed by one or more processors cause the one or more processor to: receive an input picture;
generate a lowered-resolution picture from the received input picture;
generate input encoded codes by encoding the lowered-resolution picture;
provide the input encoded codes into a super-resolution model to generate, as prediction encoded codes, encoded codes of a high-resolution picture corresponding to the input encoded codes; and
train the super-resolution model based on a loss function defined as a combination of a first difference and a second difference, the first difference representing a difference between the input picture and a decoded version of the prediction encoded codes, and the second difference representing a difference between encoded codes of the input picture and the prediction encoded codes.
5 . A method for super-resolution imaging by using a pre-trained super-resolution model to perform super-resolution imaging at a code level, the method comprising:
obtaining, as subject encoded codes, encoded codes of a subject picture;
providing the subject encoded codes into the pre-trained super-resolution model to generate prediction encoded codes, wherein the pre-trained super-resolution model is obtained by iteratively training based on a loss function defined as a combination of a first difference and a second difference, the first difference representing a difference between an input picture and a decoded version of encoded codes predicted by an untrained super-resolution model, and the second difference representing a difference between encoded codes of the input picture and the encoded codes predicted by the untrained super-resolution model;
decoding the prediction encoded codes to generate decoded prediction codes; and
generating a version of the subject picture with increased resolution based on the decoded prediction codes.
6 . The method of claim 5 , wherein the super-resolution model is trained further by:
receiving the input picture;
generating a lowered-resolution picture from the received input picture;
generating input encoded codes by encoding the lowered-resolution picture; and
providing the input encoded codes into the untrained super-resolution model to generate, as prediction encoded codes, encoded codes of a high-resolution picture corresponding to the input encoded codes.
7 . The method of claim 5 , wherein the subject encoded codes are generated by entropy decoding and de-quantizing a compressed picture.
8 . The method of claim 7 , wherein inverted discrete cosine transform (DCT) is performed on the prediction encoded codes to generate the version of the subject picture with increased resolution.