Sparse light field representation
The disclosure provides an approach for generating a sparse representation of a light field. In one configuration, a sparse representation application receives a light field constructed from multiple images, and samples and stores a set of line segments originating at various locations in epipolar-plane images (EPI), until the EPIs are entirely represented and redundancy is eliminated to the extent possible. In addition, the sparse representation application determines and stores difference EPIs that account for variations in the light field. Taken together, the line segments and the difference EPIs compactly store all relevant information that is necessary to reconstruct the full 3D light field and extract an arbitrary input image with a corresponding depth map, or a full 3D point cloud, among other things. This concept also generalizes to higher dimensions. In a 4D light field, for example, the principles of eliminating redundancy and storing a difference volume remain valid.
1. A computer-implemented method for generating and storing a compact representation of a light field, comprising:
receiving the light field captured as a plurality of images depicting a scene, wherein the light field is one of a three-dimensional (3D) light field and a four-dimensional (4D) light field;
receiving depth estimates of points in the scene;
determining an error between a reconstruction from the depth estimates and the received images; and
storing, in computer storage hardware, the compact representation of the light field which includes the depth estimates and the determined error,
wherein, when the light field is the 3D light field, the depth estimates include slopes of lines at pixels of epipolar-plane images (EPIs) generated from the plurality of images, the reconstruction includes one or more EPIs generated based on the slopes, and the error is an error between the EPIs generated from the plurality of images and the one or more reconstructed EPIs, and
wherein, when the light field is the 4D light field, the depth estimates include planes passing through pixels of 3D epipolar (plane) volumes generated from the plurality of images, the reconstruction includes one or more 3D epipolar (plane) volumes generated based on the planes, and the error is an error between the 3D epipolar (plane) volumes generated from the plurality of images and the one or more reconstructed 3D epipolar (plane) volumes.
2. The method of claim 1 , wherein, when the light field is the 3D light field, the depth estimates are stored as tuples each representing a respective one of the lines and having form l=(m, u, s, r t ), where m is a slope, (u, s) is a point, and r is an average color of the point (u, s) in one of the EPIs.
3. The method of claim 2 , wherein the tuples are stored in order of decreasing slope.
4. The method of claim 1 , wherein, when the light field is the 4D light field, the depth estimates are stored as tuples l each representing a respective one of the planes.
5. The method of claim 1 , wherein the light field further includes an unstructured light field or unstructured set of images, and wherein the depth estimates further include one of a depth map, a disparity map, and a 3D representation or model of the scene associated with the unstructured light field.
6. The method of claim 1 , wherein, when the light field is the 3D light field, the received depth estimates of points in the scene are estimated by:
generating the EPIs from the plurality of images;
iteratively downsampling the EPIs to coarser resolutions; and
at each of the iterations, for each of the EPIs:
determining edge confidence scores for pixels of the EPI,
determining depth estimates for the pixels of the EPI associated with edge confidence scores above a threshold value, and
propagating the depth estimates to other pixels of the EPI.
7. The method of claim 1 , further comprising, reconstructing the light field based on the stored depth estimates and the stored error.
8. The method of claim 1 , further comprising, extracting an image and a corresponding depth map, or a full point cloud, based on the stored depth estimates and the stored error.
9. A non-transitory computer-readable storage medium storing instructions, which when executed by a computer system, perform operations for generating and storing a compact representation of a light field, the operations comprising:
receiving the light field captured as a plurality of images depicting a scene, wherein the light field is one of a three-dimensional (3D) light field and a four -dimensional (4D) light field;
receiving depth estimates of points in the scene;
determining an error between a reconstruction from the depth estimates and the received images; and
storing, in computer storage hardware, the compact representation of the light field which includes the depth estimates and the determined error,
wherein, when the light field is the 3D light field, the depth estimates include slopes of lines at pixels of epipolar-plane images (EPIs) generated from the plurality of images, the reconstruction includes one or more EPIs generated based on the slopes, and the error is an error between the EPIs generated from the plurality of images and the one or more reconstructed EPIs, and
wherein, when the light field is the 4D light field, the depth estimates include planes passing through pixels of 3D epipolar (plane) volumes generated from the plurality of images, the reconstruction includes one or more 3D epipolar (plane) volumes generated based on the planes, and the error is an error between the 3D epipolar (plane) volumes generated from the plurality of images and the one or more reconstructed 3D epipolar (plane) volumes.
10. The computer-readable storage medium of claim 9 , wherein, when the light field is the 3D light field, the depth estimates are stored as tuples each representing a respective one of the lines and having form l=(m, u, s, r T ), where m is a slope, (u, s) is a point, and r is an average color of the point (u, s) in one of the EPIs.
11. The computer-readable storage medium of claim 10 , wherein the tuples are stored in order of decreasing slope.
12. The computer-readable storage medium of claim 9 , wherein, when the light field is the 4D light field, the depth estimates are stored as tuples l each representing a respective one of the planes.
13. The computer-readable storage medium of claim 9 , wherein the light field further includes an unstructured light field or unstructured set of images, and wherein the depth estimates further include one of a depth map, a disparity map, and a 3D representation or model of the scene associated with the unstructured light field.
14. The computer-readable storage medium of claim 9 , wherein, when the light field is the 3D light field, the received depth estimates of points in the scene are estimated by:
generating the EPIs from the plurality of images;
iteratively downsampling the EPIs to coarser resolutions; and
at each of the iterations, for each of the EPIs:
determining edge confidence scores for pixels of the EPI,
determining depth estimates for the pixels of the EPI associated with edge confidence scores above a threshold value, and
propagating the depth estimates to other pixels of the EPI.
15. The computer-readable storage medium of claim 9 , further comprising one of reconstructing the light field based on the stored depth estimates and the stored error and extracting an image and a corresponding depth map, or a full point cloud, based on the stored depth estimates and the stored error.
16. A system, comprising:
a processor; and
a memory, wherein the memory includes an application program configured to perform operations for generating and storing a compact representation of a light field, the operations comprising:
receiving the light field captured as a plurality of images depicting a scene, wherein the light field is one of a three-dimensional (3D) light field and a four-dimensional (4D) light field,
receiving depth estimates of points in the scene,
determining an error between a reconstruction from the depth estimates and the received images, and
storing, in computer storage hardware, the compact representation of the light field which includes the depth estimates and the determined error,
wherein, when the light field is the 3D light field, the depth estimates include slopes of lines at pixels of epipolar-plane images (EPIs) generated from the plurality of images, the reconstruction includes one or more EPIs generated based on the slopes, and the error is an error between the EPIs generated from the plurality of images and the one or more reconstructed EPIs, and
wherein, when the light field is the 4D light field, the depth estimates include planes passing through pixels of 3D epipolar (plane) volumes generated from the plurality of images, the reconstruction includes one or more 3D epipolar (plane) volumes generated based on the planes, and the error is an error between the 3D epipolar (plane) volumes generated from the plurality of images and the one or more reconstructed 3D epipolar (plane) volumes.