Extended depth-of-field correction using reconstructed depth map
A computer-implemented method includes obtaining 3D model of real-world environment; receiving image of real-world environment captured using camera and pose information indicative of camera pose from which image is captured; utilising 3D model of real-world environment to generate reconstructed depth map from perspective of camera pose; and applying extended depth-of-field correction to image segment(s) of image that is/are out of focus, by using point spread function determined for camera, based on optical depths in segment(s) of reconstructed depth map corresponding to image segment(s) of image.
1. A computer-implemented method comprising:
obtaining a three-dimensional (3D) model of a real-world environment;
receiving a given image of the real-world environment captured using a given camera, and pose information indicative of a given camera pose from which the given image is captured;
utilising the 3D model of the real-world environment to generate a reconstructed depth map from a perspective of the given camera pose;
applying an extended depth-of-field correction to at least one image segment of the given image that is out of focus, by using a point spread function determined for the given camera, based on optical depths in at least one segment of the reconstructed depth map corresponding to the at least one image segment of the given image; and
obtaining the point spread function determined for the given camera;
wherein the step of obtaining the point spread function comprises:
utilising the 3D model of the real-world environment to generate a reconstructed image from a perspective of the given camera pose; and
determining the point spread function for the given camera as a function of optical depth, based on a correlation between at least a subset of pixels of the given image that is out of focus and respective pixels of the reconstructed image, and respective optical depths of at least the subset of the pixels in the reconstructed depth map;
the computer-implemented method further comprising updating the point spread function by employing a neural network to predict a value of the point spread function for a given optical depth based on values of the point spread function for at least two optical depths that are determined based on said correlation.
2. The computer-implemented method of claim 1 , further comprising:
identifying a plurality of image segments of the given image that represent different objects or different parts of the objects that are present in the real-world environment, the plurality of image segments being out of focus;
computing weights for the plurality of image segments of the given image, wherein a weight of a given image segment is calculated based on at least one of:
a gradient of optical depth across the given image segment,
a difference in optical depth between a given object represented in the given image segment and a neighbourhood of the given object in the given image segment,
a contrast of features in an image segment of the reconstructed image corresponding to the given image segment; and
selecting at least one image segment, from amongst the plurality of image segments of the given image, based on the weights computed for the plurality of image segments of the given image, wherein the subset of the pixels of the given image belong to the selected at least one image segment.
3. The computer-implemented method of claim 1 , wherein the extended depth-of-field correction is applied by employing a Weiner filter to deconvolve the at least one image segment of the given image with the point spread function determined for the given camera.
4. The computer-implemented method of claim 1 , further comprising:
obtaining information indicative of a gaze direction of a user;
determining a gaze region in the given image, based on the gaze direction of the user; and
selecting the at least one image segment, from amongst a plurality of image segments of the given image that are out of focus, when the at least one image segment overlaps with the gaze region.
5. The computer-implemented method of claim 1 , further comprising:
utilising the 3D model of the real-world environment to generate a reconstructed image from a perspective of the given camera pose;
convolving the reconstructed image according to the point spread function determined for the given camera;
comparing pixels of the at least one image segment of the given image with respective pixels of the convolved reconstructed image to detect whether any changes have occurred in a region of the real-world environment that is represented in the at least one image segment of the given image; and
applying the extended depth-of-field correction to the at least one image segment of the given image, only when it is detected that no changes have occurred in said region of the real-world environment.
6. A system comprising at least one server configured to:
obtain a three-dimensional (3D) model of a real-world environment;
receive a given image of the real-world environment captured using a given camera, and pose information indicative of a given camera pose from which the given image is captured;
utilise the 3D model of the real-world environment to generate a reconstructed depth map from a perspective of the given camera pose;
apply an extended depth-of-field correction to at least one image segment of the given image that is out of focus, by using a point spread function determined for the given camera, based on optical depths in at least one segment of the reconstructed depth map corresponding to the at least one image segment of the given image;
wherein the at least one server is configured to obtain the point spread function determined for the given camera;
wherein when obtaining the point spread function, the at least one server is configured to:
utilise the 3D model of the real-world environment to generate a reconstructed image from a perspective of the given camera pose; and
determine the point spread function for the given camera as a function of optical depth, based on a correlation between at least a subset of pixels of the given image that is out of focus and respective pixels of the reconstructed image, and respective optical depths of at least the subset of the pixels in the reconstructed depth map;
wherein the at least one server is further configured to update the point spread function by employing a neural network to predict a value of the point spread function for a given optical depth based on values of the point spread function for at least two optical depths that are determined based on said correlation.
7. The system of claim 6 , wherein the at least one server is further configured to:
identify a plurality of image segments of the given image that represent different objects or different parts of the objects that are present in the real-world environment, the plurality of image segments being out of focus;
compute weights for the plurality of image segments of the given image, wherein a weight of a given image segment is calculated based on at least one of:
a gradient of optical depth across the given image segment,
a difference in optical depth between a given object represented in the given image segment and a neighbourhood of the given object in the given image segment,
a contrast of features in an image segment of the reconstructed image corresponding to the given image segment; and
select at least one image segment, from amongst the plurality of image segments of the given image, based on the weights computed for the plurality of image segments of the given image, wherein the subset of the pixels of the given image belong to the selected at least one image segment.
8. The system of claim 6 , wherein the at least one server is configured to apply the extended depth-of-field correction by employing a Weiner filter to deconvolve the at least one image segment of the given image with the point spread function determined for the given camera.
9. The system of claim 6 , wherein the at least one server is further configured to:
obtain information indicative of a gaze direction of a user;
determine a gaze region in the given image, based on the gaze direction of the user; and
select the at least one image segment, from amongst a plurality of image segments of the given image that are out of focus, when the at least one image segment overlaps with the gaze region.
10. The system of claim 6 , wherein the at least one server is further configured to:
utilise the 3D model of the real-world environment to generate a reconstructed image from a perspective of the given camera pose;
convolve the reconstructed image according to the point spread function determined for the given camera;
compare pixels of the at least one image segment of the given image with respective pixels of the convolved reconstructed image to detect whether any changes have occurred in a region of the real-world environment that is represented in the at least one image segment of the given image; and
apply the extended depth-of-field correction to the at least one image segment of the given image, only when it is detected that no changes have occurred in said region of the real-world environment.
11. A computer program product comprising a non-transitory machine-readable data storage medium having stored thereon program instructions that, when executed by a processor, cause the processor to execute steps of a computer-implemented method of claim 1 .