Systems and methods utilizing a machine learning model for generating defocus blur effects
In one embodiment, a system may access a training sample that includes training images and corresponding training depth maps of a scene, with the training images being associated with different predetermined viewpoints of the scene. The system may generate elemental images of the scene by processing the training images and the training depth maps using a machine-learning model. The elemental images are associated with more viewpoints of the scene than the predetermined viewpoints associated with the training images. The system may update the machine-learning model based on a comparison between the generated elemental images of the scene and target elemental images that are each associated with a predetermined viewpoint. The updated machine-learning model is configured to generate elemental images of a scene of interest based on input images and corresponding depth maps of the scene of interest from different viewpoints.
1. A method comprising:
by a computing system, accessing a training sample for a machine-learning model, the training sample for the machine-learning model comprising a first set of input images and corresponding depth maps of a first scene, wherein the input images represent the first scene from a predetermined number of viewpoints;
by the computing system, generating, using the machine-learning model, a first set of elemental images of the first scene by processing the input images and the corresponding depth maps, wherein the elemental images generated using the machine-learning model represent the first scene from a number of viewpoints that is greater than the predetermined number of viewpoints associated with the input images;
by the computing system, updating the machine-learning model based on a comparison between the generated elemental images of the first scene and target elemental images associated with the training sample, wherein the target elemental images are each associated with a common viewpoint with each of the generated elemental images; and
by the computing system, generating, using the updated machine-learning model, a second set of elemental images of a second scene of interest from different viewpoints for a head-mounted display device comprising a microlens array based on a second set of input images and corresponding depth maps of the second scene of interest, wherein the generated second set of elemental images, when viewed through the microlens array of the head-mounted display device, provide a user of the head-mounted display device a view of the second scene with a synthesized defocus-blur effect.
2. The method of claim 1 , wherein the head-mounted display device is one of light field displays.
3. The method of claim 1 , wherein the predetermined number of viewpoints associated with the input images include one or more boundary viewpoints of a predetermined range of viewpoints of interest.
4. The method of claim 3 , wherein the predetermined range of viewpoints of interest corresponds to a polygonal area, wherein at least one of the one or more boundary viewpoints corresponds to a corner of the polygonal area.
5. The method of claim 3 , wherein the predetermined number of viewpoints associated with the input images include a center viewpoint in the predetermined range of viewpoints of interest.
6. The method of claim 1 , wherein the viewpoints associated with the generated elemental images of the first scene include viewpoints that are between the viewpoints associated with the input images.
7. The method of claim 1 , wherein the comparison between the generated elemental images of the first scene and the target elemental images comprises a comparison between each of the generated elemental images of the first scene and an associated one of the target elemental images.
8. The method of claim 1 , wherein the elemental images of the first scene are components of a first image and the target elemental images are components of a second image.
9. The method of claim 8 , wherein the target elemental images of the second image are arranged in a grid format based on their respective associated viewpoints.
10. The method of claim 1 ,
wherein the machine-learning model comprises at least one residual block that comprises sequentially connected layers, the sequentially connected layers comprising a first layer and a last layer; and
wherein the residual block is configured to add an input of the first layer to an output of the last layer.
11. The method of claim 1 ,
wherein the machine-learning model comprises a plurality of layers that includes a last layer and a next-to-last layer;
wherein the machine-learning model is configured to generate a concatenated result by concatenating an input of the machine-learning model with an output of the next-to-last layer; and
wherein the last layer is configured to use the concatenated result as input.
12. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
access a training sample for a machine-learning model, the training sample for the machine-learning model comprising a first set of input images and corresponding depth maps of a first scene, wherein the input images represent the first scene from a predetermined number of viewpoints;
generate, using the machine-learning model, a first set of elemental images of the first scene by processing the input images and the corresponding depth maps, wherein the elemental images generated using the machine-learning model represent the first scene from a number of viewpoints that is greater than the predetermined number of viewpoints associated with the input images;
update the machine-learning model based on a comparison between the generated elemental images of the first scene and target elemental images associated with the training sample, wherein the target elemental images are each associated with a common viewpoint with each of the generated elemental images; and
generate, using the updated machine-learning model, a second set of elemental images of a second scene of interest from different viewpoints for a head-mounted display device comprising a microlens array based on a second set of input images and corresponding depth maps of the second scene of interest, wherein the generated second set of elemental images, when viewed through the microlens array of the head-mounted display device, provide a user of the head-mounted display device a view of the second scene with a synthesized defocus-blur effect.
13. The media of claim 12 , wherein the predetermined number of viewpoints associated with the input images include one or more boundary viewpoints of a predetermined range of viewpoints of interest.
14. The media of claim 13 , wherein the predetermined range of viewpoints of interest corresponds to a polygonal area, wherein at least one of the one or more boundary viewpoints corresponds to a corner of the polygonal area.
15. The media of claim 13 , wherein the predetermined number of viewpoints associated with the input images include a center viewpoint in the predetermined range of viewpoints of interest.
16. The media of claim 12 , wherein the viewpoints associated with the generated elemental images of the first scene include viewpoints that are between the viewpoints associated with the input images.
17. A system comprising: one or more processors; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:
access a training sample for a machine-learning model, the training sample for the machine-learning model comprising a first set of input images and corresponding depth maps of a first scene, wherein the input images represent the first scene from a predetermined number of viewpoints;
generate, using the machine-learning model, a first set of elemental images of the first scene by processing the input images and the corresponding depth maps, wherein the elemental images generated using the machine-learning model represent the first scene from a number of viewpoints that is greater than the predetermined number of viewpoints associated with the input images;
update the machine-learning model based on a comparison between the generated elemental images of the first scene and target elemental images associated with the training sample, wherein the target elemental images are each associated with a common viewpoint with each of the generated elemental images; and
generate, using the updated machine-learning model, a second set of elemental images of a second scene of interest from different viewpoints for a head-mounted display device comprising a microlens array based on a second set of input images and corresponding depth maps of the second scene of interest, wherein the generated second set of elemental images, when viewed through the microlens array of the head-mounted display device, provide a user of the head-mounted display device a view of the second scene with a synthesized defocus-blur effect.
18. The system of claim 17 , wherein the predetermined number of viewpoints associated with the input images include one or more boundary viewpoints of a predetermined range of viewpoints of interest.
19. The system of claim 18 , wherein the predetermined range of viewpoints of interest corresponds to a polygonal area, wherein at least one of the one or more boundary viewpoints corresponds to a corner of the polygonal area.
20. The system of claim 18 , wherein the predetermined number of viewpoints associated with the input images include a center viewpoint in the predetermined range of viewpoints of interest.