Systems and methods for generating defocus blur effects

1. A method for using a machine-learning model to generate images with defocus-blur effects based on in-focus input images and corresponding depth maps, the method comprising, by a computing system:

receiving current eye-tracking data associated with a user of a head-mounted display;

dynamically adjusting a focal length of the head-mounted display based on the current eye-tracking data;

generating an in-focus image of a scene and a corresponding depth map of the scene;

generating a circle-of-confusion map for the scene based on the depth map, wherein the circle-of-confusion map encodes a desired focal surface in the scene;

generating, using the machine-learning model, an output image with a synthesized defocus-blur effect by processing the in-focus image, the corresponding depth map, and the circle-of-confusion map of the scene; and

displaying the output image with the synthesized defocus-blur effect to the user via the head-mounted display having the adjusted focal length.

2. The method of claim 1 , wherein the head-mounted display is a varifocal near-eye display.

3. The method of claim 1 , wherein dynamically adjusting the focal length of the head-mounted display comprises:

expanding or contracting an eyepiece of the head-mounted display based on the current eye-tracking data.

4. The method of claim 1 , wherein each object in the scene is in-focus in the in-focus image.

5. The method of claim 1 , further comprising training the machine-learning model to generate the images with defocus-blur effects based on the in-focus input images and the corresponding depth maps.

6. The method of claim 5 , wherein training the machine-learning model comprises:

accessing training samples from a training dataset for the machine-learning model, the training samples for the machine-learning model comprising the in-focus input images of a plurality of scenes and corresponding depth maps of the scenes;

accessing circle-of-confusion maps for the plurality of scenes depicted in the in-focus input images, each of the circle-of-confusion maps encoding a desired focal surface in a particular scene of the plurality of scenes;

generating, using the machine-learning model, output images with synthesized defocus-blur effects by processing the in-focus input images, the corresponding depth maps, and the corresponding circle-of-confusion maps; and

updating the machine-learning model based on a comparison between the generated output images and target images associated with the training samples, wherein a target image depicts the particular scene with a desired defocus-blur effect.

7. The method of claim 1 ,

wherein the machine-learning model comprises at least one residual block that comprises sequentially connected layers, the sequentially connected layers comprising a first layer and a last layer; and

wherein the residual block is configured to add an input of the first layer to an output of the last layer.

8. The method of claim 1 ,

wherein the machine-learning model comprises a plurality of layers that includes a last layer and a next-to-last layer;

wherein the machine-learning model is configured to generate a concatenated result by concatenating an input of the machine-learning model with an output of the next-to-last layer; and

wherein the last layer is configured to use the concatenated result as input.

9. One or more computer-readable non-transitory storage media embodying software for using a machine-learning model to generate images with defocus-blur effects based on in-focus input images and corresponding depth maps, wherein the software is operable when executed to:

receive current eye-tracking data associated with a user of a head-mounted display;

dynamically adjust a focal length of the head-mounted display based on the current eye-tracking data;

generate an in-focus image of a scene and a corresponding depth map of the scene;

generate a circle-of-confusion map for the scene based on the depth map, wherein the circle-of-confusion map encodes a desired focal surface in the scene;

generate, using the machine-learning model, an output image with a synthesized defocus-blur effect by processing the in-focus image, the corresponding depth map, and the circle-of-confusion map of the scene; and

display the output image with the synthesized defocus-blur effect to the user via the head-mounted display having the adjusted focal length.

10. The media of claim 9 , wherein the head-mounted display is a varifocal near-eye display.

11. The media of claim 9 , wherein dynamically adjusting the focal length of the head-mounted display comprises:

expanding or contracting an eyepiece of the head-mounted display based on the current eye-tracking data.

12. The media of claim 9 , wherein each object in the scene is in-focus in the in-focus image.

13. The media of claim 9 , wherein the software is further operable when executed to:

train the machine-learning model to generate the images with defocus-blur effects based on the in-focus input images and the corresponding depth maps.

14. The media of claim 13 , wherein training the machine-learning model comprises:

accessing training samples from a training dataset for the machine-learning model, the training samples for the machine-learning model comprising the in-focus input images of a plurality of scenes and corresponding depth maps of the scenes;

accessing circle-of-confusion maps for the plurality of scenes depicted in the in-focus input images, each of the circle-of-confusion maps encoding a desired focal surface in a particular scene of the plurality of scenes;

generating, using the machine-learning model, output images with synthesized defocus-blur effects by processing the in-focus input images, the corresponding depth maps, and the corresponding circle-of-confusion maps; and

updating the machine-learning model based on a comparison between the generated output images and target images associated with the training samples, wherein a target image depicts the particular scene with a desired defocus-blur effect.

15. A system for using a machine-learning model to generate images with defocus-blur effects based on in-focus input images and corresponding depth maps, the system comprising: one or more processors; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:

receive current eye-tracking data associated with a user of a head-mounted display;

dynamically adjust a focal length of the head-mounted display based on the current eye-tracking data;

generate an in-focus image of a scene and a corresponding depth map of the scene;

generate a circle-of-confusion map for the scene based on the depth map, wherein the circle-of-confusion map encodes a desired focal surface in the scene;

generate, using the machine-learning model, an output image with a synthesized defocus-blur effect by processing the in-focus image, the corresponding depth map, and the circle-of-confusion map of the scene; and

display the output image with the synthesized defocus-blur effect to the user via the head-mounted display having the adjusted focal length.

16. The system of claim 15 , wherein the head-mounted display is a varifocal near-eye display.

17. The system of claim 15 , wherein dynamically adjusting the focal length of the head-mounted display comprises:

expanding or contracting an eyepiece of the head-mounted display based on the current eye-tracking data.

18. The system of claim 15 , wherein each object in the scene is in-focus in the in-focus image.

19. The system of claim 15 , wherein the processors are further operable when executing the instructions to:

train the machine-learning model to generate the images with defocus-blur effects based on the in-focus input images and the corresponding depth maps.

20. The system of claim 19 , wherein training the machine-learning model comprises:

accessing training samples from a training dataset for the machine-learning model, the training samples for the machine-learning model comprising the in-focus input images of a plurality of scenes and corresponding depth maps of the scenes;

accessing circle-of-confusion maps for the plurality of scenes depicted in the in-focus input images, each of the circle-of-confusion maps encoding a desired focal surface in a particular scene of the plurality of scenes;

generating, using the machine-learning model, output images with synthesized defocus-blur effects by processing the in-focus input images, the corresponding depth maps, and the corresponding circle-of-confusion maps; and

updating the machine-learning model based on a comparison between the generated output images and target images associated with the training samples, wherein a target image depicts the particular scene with a desired defocus-blur effect.