Passive and single-viewpoint 3D imaging system

1. A method for a passive single-viewpoint 3D imaging system, comprising:

capturing an image from a camera, the camera having one or more phase masks;

using a reconstruction algorithm, for estimation of a 3D or depth image,

wherein the reconstruction algorithm comprises:

building a differentiable forward model with an optical layer and a reconstruction network,

wherein in the optical layer, a physics-based model simulates a plurality of coded images captured on a sensor with a phase mask,

wherein in the reconstruction network, a neural network produces a depth map from the coded images, and

wherein the optical layer takes in an all-in-focus image and the corresponding depth map as inputs and outputs a physically accurate coded intensity image.

2. The method of claim 1 , wherein the phase masks are separately fabricated and inserted between a camera lens and an image sensor.

3. The method of claim 1 wherein the phase masks are integrated into the camera lens assembly and fabricated as a single optimized modified optical assembly.

4. The method of claim 1 , wherein the phase masks are integrated into the camera lens assembly and fabricated as either single or multiple optimized modified optical assembly.

5. The method of claim 1 , wherein a plurality of parameters of the neural network are trained using a red green blue depth (RGBD) dataset to optimize a height map of the phase mask and parameters in the neural network.

6. The method in claim 5 , wherein the training process comprises:

calibrating a plurality of point spread functions (PSFs) at different depths;

simulating the plurality of coded images with the calibrated PSFs to fine-tune the neural network; and

recovering the corresponding depth map by capturing and passing a plurality of images through the neural network, wherein the neural network is a U-Net.

7. The method of claim 1 , wherein the reconstruction algorithm is based on optimization of an energy function and

wherein the reconstruction algorithm returns both the 3D or depth image and a 2D intensity image.

8. The method of claim 5 , further comprising optimizing, both parameters of the optical layer and the reconstruction network, based on a loss defined between an estimated depth and a ground truth depth.

9. The method of claim 1 , further comprising fabricating the optimized phase mask using photolithography method.

10. The method of claim 9 , further comprising printing the optimized phase mask on a glass substrate and cutting the glass substrate to a smaller size to fit into the aperture plane of the camera.

11. A system for a passive and single-viewpoint 3D imaging system, comprising:

a camera; and

an optimized phase mask placed in an aperture plane of the camera,

wherein the optimized phase mask and a reconstruction algorithm allow the system to estimate a 3D or depth image,

wherein the reconstruction algorithm comprises a computer processor configured to:

build a differentiable forward model with an optical layer comprising a physics-based model to simulate a plurality of coded images captured on the sensor with the phase mask and a neural network to produce a depth map from the coded images;

train a framework with a synthetic red green blue depth (RGBD) dataset to optimize a height map of the phase mask and parameters in the neural network;

calibrate a plurality of point spread functions (PSFs) at different depths;

simulate the plurality of coded images with the calibrated PSFs to fine-tune the neural network; and

recover the corresponding depth map by capturing and passing a plurality of images through the neural network.

12. The system of claim 11 , wherein the optimized phase mask is fabricated using a 3D printer using photolithography.

13. The system of claim 11 , wherein the optical layer takes in an all-in-focus image and the corresponding depth map as inputs and outputs a physically accurate coded intensity image.

14. The system of claim 11 , wherein both parameters of the optical layer and the reconstruction network are optimized based on a loss defined between an estimated depth and a ground truth depth.

15. The system of claim 11 , wherein the optimized phase mask is printed on a glass substrate and the glass substrate cut to a smaller size to fit into the aperture plane of the camera.

16. A non-transitory computer readable medium storing instructions, the instructions executable by a computer processor and comprising functionality for:

building a differentiable forward model with an optical layer and a reconstruction network,

wherein in the optical layer, a physics-based model simulates a plurality of coded images captured on a sensor with a phase mask, and

wherein in the reconstruction network, a neural network produces a depth map from the coded images;

training a framework with a red green blue depth (RGBD) dataset to optimize a height map of the phase mask and parameters in the neural network;

fabricating an optimal phase mask using photolithography method and placing in an aperture plane of a camera;

calibrating a plurality of point spread functions (PSFs) at different depths;

simulating the plurality of coded images with the calibrated PSFs to fine-tune the neural network; and

recovering the corresponding depth map by capturing and passing a plurality of images through the neural network.

17. The non-transitory computer readable medium of claim 16 , the instructions further comprising functionality for optimizing, both parameters of the optical layer and the reconstruction network, based on a loss defined between an estimated depth and a ground truth depth.