Real-time photorealistic 3D holography with deep neural networks
A method for producing a hologram representative of a subject three-dimensional scene includes receiving and storing input digital data characterizing a first image of the subject three-dimensional scene. The method further includes processing the data in a neural network that has been trained to transform the input digital data into a holographic representation of the subject three-dimensional scene, the representation containing phase information characterizing depth and parallax of the scene. The method also includes providing an output of the holographic representation of the subject three-dimensional scene.
1 . A method for producing a hologram representative of a subject three-dimensional scene, the method comprising:
receiving and storing input digital data characterizing a first image of the subject three-dimensional scene;
processing the data in a neural network that has been trained to transform the input digital data into a holographic representation of the subject three-dimensional scene, the representation containing phase information characterizing depth and parallax of the scene; and
providing an output of the holographic representation of the subject three-dimensional scene.
2 . A method according to claim 1 , wherein the holographic representation encodes color information of the scene.
3 . A method according to claim 1 , wherein the neural network is a convolutional neural network.
4 . A method according to claim 1 , wherein the digital data characterizing the first image includes color and depth information, and the holographic representation including amplitude and phase information by color.
5 . A method according to claim 1 , wherein processing the data is configured to produce a holographic representation that reproduces occlusion effects.
6 . A method according to claim 1 , wherein the neural network has been trained to simulate Fresnel diffraction.
7 . A method according to claim 6 , wherein the neural network has been additionally trained to cause the holographic representation to exhibit a desired depth of field effect.
8 . A method according to claim 1 , wherein the neural network has been trained on training data representing scenes constructed from samples of random three-dimensional meshes having random textures.
9 . A method according to claim 8 , wherein the training data are configured to follow a probability density function in such a manner as to have a uniform pixel distribution across a range of depths.
10 . A method according to claim 1 , further comprising, before providing the output of the holographic representation, performing anti-aliasing processing of the holographic representation to render it displayable with reduced artifacts.
11 . A method for producing a hologram representative of a subject three-dimensional scene, the method comprising:
receiving and storing input digital data characterizing a set of images of the subject three-dimensional scene;
processing the data in a neural network that has been trained to transform the input digital data into a phase-only holographic representation of the subject three-dimensional scene, the representation containing phase information characterizing depth and parallax of the scene, wherein the representation is computed to take into account effects of wave-based occlusion with respect to the scene; and
providing an output of the holographic representation of the subject three-dimensional scene.
12 . A method according to claim 11 , wherein the holographic representation encodes color information of the scene.
13 . A method according to claim 11 , wherein the neural network is a convolutional neural network.
14 . A method according to claim 11 , wherein processing the data further includes performing aberration correction.
15 . A method according to claim 11 , wherein the neural network has been additionally trained to cause the holographic representation to be focused on any desired focal plane within the subject three-dimensional scene so as to exhibit a desired depth of field.
16 . A method according to claim 11 ,
wherein the neural network has received additional training in two stages to directly optimize the phase-only hologram with anti-aliasing processing by incorporating a complex to phase-only conversion into the training,
wherein in a first stage the neural network is trained to predict a midpoint hologram propagated to a center of the subject three-dimensional scene and to minimize a difference between a target focal stack and a predicted focal stack, and
in a second stage a phase-only target hologram is generated from the predicted midpoint hologram and refined by calculating a dynamic focal stack loss, between a post-encoding focal stack and the target focal stack, and a regularization loss associated therewith.
17 . A method according to claim 11 , wherein the digital data characterize a stream of images occurring at an average frame rate, and the processing is configured to occur in real time.
18 . A method according to claim 11 , wherein the set of images of the subject three-dimensional scene includes a plurality of layered depth images.
19 . A method according to claim 11 , wherein the set of images of the subject three-dimensional scene is a single RGB-D image.
20 . A method for producing a hologram representative of a subject three-dimensional scene, the method comprising:
receiving and storing input digital data characterizing a set of images of the subject three-dimensional scene;
processing the data in a neural network that has been trained to transform the input digital data into a phase-only holographic representation of the subject three-dimensional scene and has been further trained in two stages to directly optimize the phase-only hologram with anti-aliasing processing by incorporating a complex to phase-only conversion into the training,
wherein in a first stage the neural network is trained to predict a midpoint hologram propagated to a center of the subject three-dimensional scene and to minimize a difference between a target focal stack and a predicted focal stack, and
in a second stage a phase-only target hologram is generated from the predicted midpoint hologram and refined by calculating a dynamic focal stack loss, between a post-encoding focal stack and the target focal stack, and a regularization loss associated therewith, the representation containing phase information characterizing depth and parallax of the scene; and
providing an output of the holographic representation of the subject three-dimensional scene.
21 . The method of claim 20 , wherein the post-encoding focal stack is generated by applying a differentiable wave propagation operator to the phase-only target hologram at a plurality of focal planes, and parameters of the neural network are updated based on the dynamic focal stack loss computed from the generated focal stack.