Image restoration for through-display imaging
Examples are disclosed that relate to the restoration of degraded images acquired via a behind-display camera. One example provides a method of training a machine learning model, the method comprising inputting training image pairs into the machine learning model, each training image pair comprising an undegraded image and a degraded image that represents an appearance of the undegraded image to a behind-display camera, and training the machine learning model using the training image pairs to generate frequency information that is missing from the degraded images.
1. A method of training a machine learning model, the method comprising:
inputting training image pairs into the machine learning model, the machine learning model comprising a U-shaped neural network comprising a first sub-encoder and a second sub-encoder in parallel with the first sub-encoder, each training image pair comprising an undegraded image and a degraded image that represents an appearance of the undegraded image to a behind-display camera, the degraded image comprising loss of mid-frequency information due to diffraction-related blurring, the mid-frequency information within a range of 2 to 8 cycles per degree; and
training the machine learning model using the training image pairs to generate mid-frequency information to correct for the loss of mid-frequency information due to the diffraction-related blurring, wherein the first sub-encoder is trained to compute residual details and the second sub-encoder is trained to learn content encoding.
2. The method of claim 1 , wherein providing training image pairs to the machine learning model comprises providing the image pairs to a convolutional neural network.
3. The method of claim 1 , wherein the undegraded training image comprises an average of a plurality of repeated captured frames.
4. The method of claim 1 , further comprising acquiring each degraded image via a camera positioned behind a mask.
5. The method of claim 1 , wherein the degraded image comprises an image acquired via a camera array.
6. The method of claim 1 , wherein the training image pair is a part of three or more corresponding training images.
7. The method of claim 1 , further comprising, after training the machine learning model, implementing the machine learning model in a computing device for imaging via a behind-display camera.