Training a restoration model for balanced generation and reconstruction
Systems and methods for training a restoration model can leverage training for two sub-tasks to train the restoration model to generate realistic and identity-preserved outputs. The systems and methods can balance the training of the generation task and the reconstruction task to ensure the generated outputs preserve the identity of the original subject while generating realistic outputs. The systems and methods can further leverage a feature quantization model and skip connections to improve the model output and overall training.
1. A computing system for replacing corrupted features with stored high-quality features for image restoration, the system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining an input image, wherein the input image comprises one or more features;
processing the input image with a first model to generate a first output, wherein the first model comprises an encoder model;
processing the first output with a second model to generate a second output, wherein the second model comprises a feature quantization model, wherein the second output results from quantization of the first output by the feature quantization model, wherein the feature quantization model quantizes the one or more features to a code in a codebook and replaces the feature with a stored feature associated with the code, wherein the feature quantization model comprises the codebook, wherein the codebook comprises one or more learned feature codes, and wherein the feature quantization model replaces a corrupted feature with the stored feature of the codebook; and
processing the second output with a third model to generate a restoration output, wherein the third model comprises a decoder model, and wherein the restoration output comprises an output image.
2. The system of claim 1 , wherein the input image comprises a degraded facial image, and wherein the output image comprises one or more predicted pixels.
3. The system of claim 1 , wherein the feature quantization model generates a feature embedding based on an input, maps the input to a known feature map, and outputs the second output comprising a known feature of the known feature map.
4. The system of claim 1 , wherein the second output comprises latent encoding data, wherein the latent encoding data comprises a latent feature.
5. The system of claim 1 , wherein the input image is descriptive of one or more faces, wherein the one or more features are associated with one or more facial features descriptive of the one or more faces.
6. The system of claim 1 , further comprising a restoration model, wherein the restoration model comprises the first model, the second model, and the third model, and wherein the restoration model further comprises a plurality of skip connections that generate a plurality of skip connection outputs, wherein the plurality of skip connection outputs are processed by different decoder blocks of the decoder model to generate the restoration output.
7. The system of claim 1 , wherein the second output comprises a quantized feature determined based on an extracted feature from the first output.
8. The system of claim 1 , wherein processing the first output with the second model to generate the second output comprises:
processing the first output with a feature extractor to generate a feature vector, wherein the feature vector is a vector mapped to an embedding space;
determining a stored vector associated with an embedding space location of the feature vector, wherein the stored vector is obtained from a different image than the input image; and
outputting the second output, wherein the second output comprises the stored vector.
9. The system of claim 1 , wherein the third model comprises a linear gated feature fusion block.
10. A computer-implemented method for training an image restoration model that replaces corrupted features with stored high-quality features, the method comprising:
obtaining, by a computing system comprising one or more processors, a training image, wherein the training image comprises one or more features;
processing, by the computing system, the training image with a restoration model to generate a restoration output, wherein processing the training image with the restoration model comprises:
processing, by the computing system, the training image with an encoder model to generate encoding data;
processing, by the computing system, the encoding data with a feature quantization model to generate latent feature data, wherein the feature quantization model quantizes the one or more features to a code in a codebook and replaces the feature with a stored feature associated with the code, wherein the feature quantization model comprises the codebook, wherein the codebook comprises one or more learned feature codes, and wherein the feature quantization model replaces a corrupted feature with the stored feature of the codebook;
processing, by the computing system, the latent feature data with a decoder model to generate a restoration output;
evaluating, by the computing system, a loss function that evaluates a difference between the restoration output and the training image; and
adjusting, by the computing system, one or more parameters of the restoration model based at least in part on the loss function.
11. The method of claim 10 , wherein the feature quantization model comprises an embedding feature map, wherein the embedding feature map is associated with a feature quantization map descriptive of quantized features, and wherein processing the encoding data with the encoder model comprises:
generating, by the computing system, a feature embedding based on the encoding data, wherein the feature embedding is an embedding in an embedding space associated with the embedding feature map;
determining, by the computing system, an associated quantized embedding in the feature quantization map based on the feature embedding; and
outputting, by the computing system, the latent feature data, wherein the latent feature data comprises the associated quantized embedding.
12. The method of claim 10 , wherein the decoder model comprises a feature fusion block, wherein the feature fusion block is configured to fuse the latent feature data and decoder data.
13. One or more non-transitory computer-readable media that collectively store instructions for training an image restoration model that replaces corrupted features with stored high-quality features that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:
obtaining a training image, wherein the training image comprises one or more features;
processing the training image with a first model to generate a first output, wherein the first model comprises an encoder model;
processing the first output with a second model to generate a second output, wherein the second model comprises a feature quantization model, wherein the feature quantization model quantizes the one or more features to a code in a codebook and replaces the feature with a stored feature associated with the code, wherein the feature quantization model comprises the codebook, wherein the codebook comprises one or more learned feature codes, and wherein the feature quantization model replaces a corrupted feature with the stored feature of the codebook;
processing the second output with a third model to generate a restoration output, wherein the third model comprises a decoder model;
evaluating a loss function that evaluates a difference between the restoration output and the training image; and
adjusting one or more parameters of at least one of the first model, the second model, or the third model based at least in part on the loss function.
14. The one or more non-transitory computer-readable media of claim 13 , wherein the loss function comprises a realism loss and a reconstruction loss.
15. The one or more non-transitory computer-readable media of claim 14 , wherein the realism loss comprises a penalization term that encourages realistic output generation based on a determined realism based in part on a comparison with a real high-quality image.
16. The one or more non-transitory computer-readable media of claim 14 , wherein the reconstruction loss comprises a penalization term that penalizes divergence from the training image.
17. The one or more non-transitory computer-readable media of claim 13 , wherein the operations further comprise:
generating a low-quality image based in part on the training image; and
wherein the low-quality image is processed by the first model to generate the first output.
18. The one or more non-transitory computer-readable media of claim 13 , wherein the operations further comprise:
generating a noisy output based on the first output, wherein the noisy output comprises noise; and
wherein the noisy output is processed by the second model to generate the second output.
19. The one or more non-transitory computer-readable media of claim 13 , wherein the first output is generated based at least in part on a skip connection, wherein the training image is processed by one or more first blocks of the first model to generate the first output, and wherein the skip connection passes the first output from the one or more first blocks of the first model to the second model.