Systems and methods for generating defocus blur effects

1. A method for training a machine-learning model to generate images with defocus-blur effect based on input images and corresponding depth maps, the method comprising, by a computing system:

accessing a training sample from a training dataset for the machine-learning model, the training sample for the machine-learning model comprising an in-focus input image of a scene and a corresponding depth map of the scene;

accessing a circle-of-confusion map for the scene depicted in the in-focus input image, wherein the circle-of-confusion map is generated based on the depth map and encodes a desired focal surface in the scene;

generating, using the machine-learning model, an output image with synthesized defocus-blur effect by processing the in-focus input image, the corresponding depth map, and the corresponding circle-of-confusion map; and

updating the machine-learning model based on a comparison between the generated output image and a target image associated with the training sample, wherein the target image depicts the scene with a desired defocus-blur effect;

wherein the updated machine-learning model is configured to generate the images with defocus-blur effect based on the input images and the corresponding depth maps.

2. The method of claim 1 , wherein the updated machine-learning model is configured to generate the images for varifocal near-eye displays.

3. The method of claim 1 , wherein each object in the scene is in-focus in the in-focus input image and the corresponding depth map.

4. The method of claim 1 , wherein the target image with the desired defocus-blur effect is generated by path tracing using a predetermined focal length.

5. The method of claim 1 ,

wherein the depth map comprises a plurality of depth pixels;

wherein the circle-of-confusion map comprises a plurality of circle-of-confusion pixels that correspond to the plurality of depth pixels, respectively; and

wherein each of the circle-of-confusion pixels is computed based on (1) the corresponding depth pixel and (2) an aperture and a focal length of a virtual camera model.

6. The method of claim 1 ,

wherein the machine-learning model comprises at least one residual block that comprises sequentially connected layers, the sequentially connected layers comprising a first layer and a last layer; and

wherein the residual block is configured to add an input of the first layer to an output of the last layer.

7. The method of claim 1 ,

wherein the machine-learning model comprises a plurality of layers that includes a last layer and a next-to-last layer;

wherein the machine-learning model is configured to generate a concatenated result by concatenating an input of the machine-learning model with an output of the next-to-last layer; and

wherein the last layer is configured to use the concatenated result as input.

8. The method of claim 1 , wherein the updated machine-learning model is configured to generate the images for television monitors, cinema screens, computer monitors, mobile phones, or tablets.

9. The method of claim 1 , wherein the circle-of-confusion map is generated based on a circular defocus pattern.

10. The method of claim 1 , wherein the circle-of-confusion map is generated based on a defocus pattern that is different from circular.

11. The method of claim 1 , wherein the in-focus input image comprises color information from at least a first color channel and a second color channel, wherein the circle-of-confusion map is associated with the first color channel, wherein the output image is generated by further processing, using the machine-learning model, a second circle-of-confusion map associated with the second color channel.

12. The method of claim 11 , further comprising:

generating a first output using at least the color information from the first color channel and the circle-of-confusion map associated with the first color channel;

generating a second output using at least the color information from the second color channel and the second circle-of-confusion map associated with the second color channel; and

combining at least the first output and the second output to generate the output image.

13. One or more computer-readable non-transitory storage media embodying software for training a machine-learning model to generate images with defocus-blur effect based on input images and corresponding depth maps, wherein the software that is operable when executed to:

access a training sample from a training dataset for the machine-learning model, the training sample for the machine-learning model comprising an in-focus input image of a scene and a corresponding depth map of the scene;

access a circle-of-confusion map for the scene depicted in the in-focus input image, wherein the circle-of-confusion map is generated based on the depth map and encodes a desired focal surface in the scene;

generate, using the machine-learning model, an output image with synthesized defocus-blur effect by processing the in-focus input image, the corresponding depth map, and the corresponding circle-of-confusion map; and

update the machine-learning model based on a comparison between the generated output image and a target image associated with the training sample, wherein the target image depicts the scene with a desired defocus-blur effect;

wherein the updated machine-learning model is configured to generate the images with defocus-blur effect based on the input images and the corresponding depth maps.

14. The media of claim 13 ,

wherein the depth map comprises a plurality of depth pixels;

wherein the circle-of-confusion map comprises a plurality of circle-of-confusion pixels that correspond to the plurality of depth pixels, respectively; and

wherein each of the circle-of-confusion pixels is computed based on (1) the corresponding depth pixel and (2) an aperture and a focal length of a virtual camera model.

15. The media of claim 13 , wherein the training sample comprises one or more additional in-focus input images of the scene used for updating the machine-learning model, wherein color information of each of the in-focus input image and the one or more additional in-focus input images is associated with a single color channel, wherein the single color channels associated with the in-focus input image and the one or more additional in-focus input images are different.

16. The media of claim 15 , wherein the software is further operable when executed to:

access one or more additional circle-of-confusion maps associated with the one or more additional in-focus input images, respectively; and

generate one or more additional output images by processing at least the one or more additional in-focus input images and the one or more additional circle-of-confusion maps using the machine-learning model;

wherein the updating of the machine-learning model is further based on the generated one or more additional output images.

17. A system for training a machine-learning model to generate images with defocus-blur effect based on input images and corresponding depth maps, the system comprising: one or more processors; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:

access a training sample from a training dataset for the machine-learning model, the training sample for the machine-learning model comprising an in-focus input image of a scene and a corresponding depth map of the scene;

access a circle-of-confusion map for the scene depicted in the in-focus input image, wherein the circle-of-confusion map is generated based on the depth map and encodes a desired focal surface in the scene;

generate, using the machine-learning model, an output image with synthesized defocus-blur effect by processing the in-focus input image, the corresponding depth map, and the corresponding circle-of-confusion map using a machine learning model; and

update the machine-learning model based on a comparison between the generated output image and a target image associated with the training sample, wherein the target image depicts the scene with a desired defocus-blur effect;

wherein the updated machine-learning model is configured to generate the images with defocus-blur effect based on the input images and the corresponding depth maps.

18. The system of claim 17 ,

wherein the depth map comprises a plurality of depth pixels;

wherein the circle-of-confusion map comprises a plurality of circle-of-confusion pixels that correspond to the plurality of depth pixels, respectively; and

wherein each of the circle-of-confusion pixels is computed based on (1) the corresponding depth pixel and (2) an aperture and a focal length of a virtual camera model.

19. The system of claim 17 , wherein the training sample comprises one or more additional in-focus input images of the scene used for updating the machine-learning model, wherein color information of each of the in-focus input image and the one or more additional in-focus input images is associated with a single color channel, wherein the single color channels associated with the in-focus input image and the one or more additional in-focus input images are different.

20. The system of claim 19 , wherein the processors are further operable when executing the instructions to:

access one or more additional circle-of-confusion maps associated with the one or more additional in-focus input images, respectively; and

generate one or more additional output images by processing at least the one or more additional in-focus input images and the one or more additional circle-of-confusion maps using the machine-learning model;

wherein the updating of the machine-learning model is further based on the generated one or more additional output images.