Image restoration for through-display imaging
Examples are disclosed that relate to the restoration of degraded images acquired via a behind-display camera. One example provides a method of training a machine learning model, the method comprising inputting training image pairs into the machine learning model, each training image pair comprising an undegraded image and a degraded image that represents an appearance of the undegraded image to a behind-display camera, and training the machine learning model using the training image pairs to generate frequency information that is missing from the degraded images.
1. On a computing device, a method of obtaining a restored image, the method comprising:
acquiring a plurality of degraded images via a plurality of behind display cameras, each degraded image of the plurality of degraded images comprising missing frequency information in a frequency region due to having been acquired through a display, each degraded image also comprising disparities due to spatial separation of the plurality of behind display cameras;
providing the plurality of degraded images as input to a machine learning model; and
receiving an output from the machine learning model, the output comprising a restored image comprising generated frequency information in the frequency region, the restored image also comprising disparity corrections.
2. The method of claim 1 wherein the plurality of behind display cameras comprises one or more wider angle cameras and one or more narrower angle cameras;
wherein the plurality of degraded images comprises one or more degraded images acquired via the one or more wider angle cameras; and
wherein the restored combined image comprises mitigation of diffracted light arising from a light source within the field of view of the wider angle camera, the light source being outside a field of view of the one or more narrower angle cameras.
3. The method of claim 1 , wherein acquiring a plurality of degraded image comprises acquiring one or more degraded images from a behind display camera with a first color filter and one or more degraded images from a camera with a second color filter.
4. The method of claim 3 , wherein the plurality of behind display cameras further comprises a clear channel camera configured to accept all visible wavelengths.
5. The method of claim 1 , wherein providing the plurality of degraded images as input to the machine learning model comprises providing the plurality of degraded images as input to a convolutional neural network.
6. The method of claim 1 , wherein providing the plurality of degraded images as input to the machine learning model comprises providing the plurality of degraded images as input to a U-shaped neural network.
7. The method of claim 6 , wherein the U-shaped neural network comprises two or more sub-encoders.
8. A computing device, comprising:
a camera array comprising a plurality of behind display cameras;
a logic subsystem; and
a storage subsystem storing instructions executable by the logic subsystem to:
for each behind display camera of the camera array, receive a degraded image comprising missing frequency information in a frequency region due to having been acquired through a display, the degraded image also comprising disparities due to a spatial separation of the behind display camera relative to other behind display cameras of the camera array,
after receiving a degraded image from each behind display camera, form a collection of degraded images comprising each degraded image and information associating each degraded image with a behind display camera of the camera array;
provide the collection of degraded images as input into a machine learning model, and
receive an output from the machine learning model of a restored combined image, the restored combined image comprising generated frequency information in the frequency region, the restored combined image further comprising disparity corrections.
9. The computing device of claim 8 , wherein the camera array comprises a plurality of separate camera modules.
10. The computing device of claim 9 , wherein the camera array comprises an array of camera lenses formed in an integrated block, the array of camera lenses configured to form images on a same image sensor.
11. The computing device of claim 8 , wherein the camera array comprises one or more wider angle behind display cameras and one or more narrower angle behind display cameras;
wherein each of the one or more wider angle cameras and one or more narrower angle cameras is behind a display comprising slits with slit width a;
wherein each of the narrower angle cameras has a field of view equal to θ 0 ; and
wherein the wider angle behind display camera comprises a field of view of at least θ 0 +θ max selected to capture all light that has relative diffraction efficiency greater than R for wavelength λ, θ max satisfying
R
=
sinc
2
(
a
π
λ
sinθ
max
)
.
12. The computing device of claim 8 , wherein the camera array comprises one or more wider angle behind display cameras and one or more narrower angle behind display cameras;
wherein each of the one or more narrower angle cameras comprises a horizontal field of view between 30° and 35° and a vertical field of view between 23° and 28°; and
wherein each of the one or more wider angle behind display cameras comprises a horizontal field of view between 71° and 81° and a vertical field of view between 64° and 74°.
13. The computing device of claim 8 , wherein the plurality of behind display cameras comprises a first color filter and a second color filter.
14. The computing device of claim 13 , wherein the plurality of behind display cameras further comprises a clear channel camera configured to accept all visible wavelengths.
15. The computing device of claim 8 , wherein the machine learning model comprises a convolutional neural network.
16. The computing device of claim 8 , wherein the machine learning model comprises a U-shaped network.