Generating super-resolution training data with pairings of original and derivative images
Systems and methods are provided for obtaining and using training data for training a super-resolution model that transforms images from a first-low resolution to a second-high resolution. The methods include generating high-fidelity high-resolution images for the training data from underlying images at the second-high resolution and that have a relatively lower fidelity. The high-fidelity high-resolution images are paired with correlating low-resolution images and used to train the super-resolution model.
1 . A method for generating training data for training a super-resolution model, the method comprising:
identifying a first super-resolution model that is to be trained to transform low-resolution input images having a first resolution into high-resolution output images having a second resolution that is higher than the first resolution;
identifying a second super-resolution model trained to transform high-resolution input images having the second resolution into super-high-resolution output images having a third resolution that is higher than the second resolution;
obtaining an initial set of high-resolution images having the second resolution;
obtaining a correlating set of low-resolution images that correlate with the initial set of high-resolution images;
applying the second super-resolution model to the initial set of high-resolution images to obtain a set of super-high-resolution images having the third resolution;
generating a set of high-fidelity high-resolution images having the second resolution by downsampling the set of super-high-resolution images from the third resolution to the second resolution, the set of high-fidelity high-resolution images having a second image fidelity that is greater than a first image fidelity of the initial set of high-resolution images; and
pairing the set of high-fidelity high-resolution images with the correlating set of low-resolution images as training data for training the super-resolution model.
2 . The method of claim 1 , wherein the method further includes applying the first super-resolution model to the training data to generate a trained super-resolution model that is configured to transform new input images having the first resolution into new output images having the second resolution with the second image fidelity.
3 . The method of claim 2 , wherein the method further includes applying anti-aliasing to the high-fidelity high-resolution images prior to applying the first super-resolution model to the training data.
4 . The method of claim 1 , wherein the correlating set of low-resolution images having the first resolution is obtained by downsampling the initial set of high-resolution images from the second resolution to the first resolution.
5 . The method of claim 1 , the correlating set of low-resolution images having the first resolution is obtained from an image generator that generates the initial set of high-resolution images.
6 . The method of claim 5 , wherein the image generator is a game engine and wherein the initial set of high-resolution images and the correlating set of low-resolution images are obtained from different viewports of the game engine.
7 . The method of claim 1 , wherein the initial set of high-resolution images having the second resolution is obtained by transforming a different set of images having a fourth resolution into the initial set of high-resolution images having the second resolution.
8 . The method of claim 7 , wherein the different set of images having the fourth resolution is transformed into the initial set of high-resolution images having the second resolution by downsampling the different set of images from the fourth resolution to the second resolution.
9 . The method of claim 7 , wherein the different set of images having the fourth resolution is transformed into the initial set of high-resolution images having the second resolution by upsampling the different set of images from the fourth resolution to the second resolution.
10 . The method of claim 1 , wherein the third resolution is at least double the second resolution.
11 . A computing system comprising:
a hardware processing system comprising a hardware processor; and
one or more storage devices storing executable instructions that are executed by the hardware processing system for causing the computing system to perform operations comprising:
identifying a first super-resolution model that is to be trained to transform low-resolution input images having a first resolution into high-resolution output images having a second resolution that is higher than the first resolution;
identifying a second super-resolution model trained to transform high-resolution input images having the second resolution into super-high-resolution output images having a third resolution that is higher than the second resolution;
obtaining an initial set of high-resolution images having the second resolution;
obtaining a correlating set of low-resolution images that correlate with the initial set of high-resolution images;
applying the second super-resolution model to the initial set of high-resolution images to obtain a set of super-high-resolution images having the third resolution;
generating a set of high-fidelity high-resolution images having the second resolution by downsampling the set of super-high-resolution images from the third resolution to the second resolution, the set of high-fidelity high-resolution images having a second image fidelity that is greater than a first image fidelity of the initial set of high-resolution images; and
pairing the set of high-fidelity high-resolution images with the correlating set of low-resolution images as training data for training the super-resolution model.
12 . The computing system of claim 11 , wherein the operations further include applying the first super-resolution model to the training data to generate a trained super-resolution model that is configured to transform new input images having the first resolution into new output images having the second resolution with the second image fidelity.
13 . The computing system of claim 12 , wherein the operations further include applying anti-aliasing to the high-fidelity high-resolution images prior to applying the first super-resolution model to the training data.
14 . The computing system of claim 11 , wherein the correlating set of low-resolution images having the first resolution is obtained by downsampling the initial set of high-resolution images from the second resolution to the first resolution.
15 . The computing system of claim 11 , the correlating set of low-resolution images having the first resolution is obtained from an image generator that generates the initial set of high-resolution images.
16 . The computing system of claim 15 , wherein the image generator is a game engine and wherein the initial set of high-resolution images and the correlating set of low-resolution images are obtained from different viewports of the game engine.
17 . The computing system of claim 11 , wherein the initial set of high-resolution images having the second resolution is obtained by transforming a different set of images having a fourth resolution into the initial set of high-resolution images having the second resolution.
18 . The computing system of claim 17 , wherein the different set of images having the fourth resolution is transformed into the initial set of high-resolution images having the second resolution by downsampling the different set of images from the fourth resolution to the second resolution.
19 . The computing system of claim 17 , wherein the different set of images having the fourth resolution is transformed into the initial set of high-resolution images having the second resolution by upsampling the different set of images from the fourth resolution to the second resolution.
20 . The computing system of claim 11 , wherein the third resolution is at least double the second resolution.