Generating images from light fields utilizing virtual viewpoints
View Patent ↗Systems and methods for the synthesis of light field images from virtual viewpoints in accordance with embodiments of the invention are disclosed. In one embodiment of the invention, a system includes a processor and a memory configured to store captured light field image data and an image manipulation application, wherein the captured light field image data includes image data, pixel position data, and a depth map, and wherein the image manipulation application configures the processor to obtain captured light field image data, determine a virtual viewpoint for the captured light field image data, where the virtual viewpoint includes a virtual location and virtual depth information, compute a virtual depth map based on the captured light field image data and the virtual viewpoint, and generate an image from the perspective of the virtual viewpoint based on the captured light field image data and the virtual depth map.
1. A system configured to synthesize images using a plurality of images captured from multiple viewpoints, comprising:
a processor; and
a memory connected to the processor and configured to store the plurality of images captured from the multiple viewpoints and an image manipulation application;
wherein the plurality of images comprises:
image data and pixel position data, the plurality of images being captured using an array camera comprising a plurality of cameras that capture the plurality of images from viewpoints of the plurality of cameras, wherein multiple cameras in the array camera simultaneously capture the plurality of images, the viewpoints of the plurality of cameras including a viewpoint of a first camera within the array camera and additional viewpoints, the additional viewpoints including a viewpoint of a second camera within the array camera, and wherein the plurality of images includes occluded pixel information captured from at least one of the additional viewpoints describing pixels not visible from the viewpoint of the first camera; and
wherein the image manipulation application configures the processor to:
obtain the plurality of images;
generate a depth map from the viewpoint of the first camera for the plurality of images using the plurality of images, where the depth map comprises depth information for one or more pixels in the image data, wherein generating the depth map comprises:
(1) for each of a plurality of depth levels, shifting the plurality of images into a stack of images for a particular depth level and computing a variance for a particular pixel in the image stack; and
(2) determining a depth level for the particular pixel by minimizing the variance for the particular pixel across the image stack;
determine a virtual viewpoint for the plurality of images based on the pixel position data and the depth map for the plurality of images, where the virtual viewpoint comprises a virtual location and virtual depth information, wherein the virtual viewpoint is a separate viewpoint from the viewpoint of the first camera, and is a viewpoint from an interpolated position between the viewpoint of the first camera and the viewpoint of the second camera;
compute a virtual depth map based on the plurality of images by projecting pixel depth information from the depth map from the viewpoint of the first camera to the virtual viewpoint; and
generate an image from the perspective of the virtual viewpoint based on the plurality of images and the virtual depth map by projecting pixels from the plurality of images based on the pixel position data and the depth map, where:
the generated image comprises a plurality of pixels selected from the image data based on the pixel position data and the virtual depth map, and
the pixels that are projected include at least one occluded pixel from the occluded pixel information that is visible from the perspective of the virtual viewpoint.
2. The system of claim 1 , wherein:
at least one projected pixel in the generated image is not described in the image data, the pixel position data, and the depth map; and
the image manipulation application further configures the processor to generate the at least one projected pixel by resampling the image data, the pixel position data, and the depth map.
3. The system of claim 1 , wherein a pinhole camera model is utilized to project pixels within the generated image based on light rays projecting from the virtual viewpoint, where each projected pixel is associated with at least one of the projected light rays.
4. The system of claim 3 , wherein projected pixel depth information is determined for at least one pixel in the generated image based on the depth map, the virtual viewpoint, and light rays associated with the at least one projected pixel.
5. The system of claim 4 , wherein protected pixel depth information for a projected pixel is based on minimizing the variance for the projected pixel across the image data within the plurality of images captured from the multiple viewpoints.
6. The system of claim 3 , wherein the image manipulation application further configures the processor to combine projected pixels having the same location within the generated image.
7. The system of claim 6 , wherein the projected pixels having the same location are combined based on a weighted average of the pixels.
8. The system of claim 1 , further comprising a sensor configured to obtain input data indicative of a position within the plurality of images captured from the multiple viewpoints.
9. The system of claim 8 , wherein the sensor is a touchscreen interface.
10. The system of claim 8 , wherein the sensor is configured to obtain spatial location information.
11. The system of claim 8 , wherein the sensor is a camera configured to obtain input data selected from the group consisting of head tracking data and gaze tracking data.
12. The system of claim 8 , wherein the virtual viewpoint is selected based on the input data.