Real-time spherical light field reconstruction from a single omnidirectional image

1 . An image processing method comprising:

receiving frame data in an equirectangular projection (ERP) format from a camera that captures omnidirectional images of an imaged scene;

for each of at least one of the omnidirectional images, converting the frame data into a cubemap projection (CMP) format;

in a reconstruction network that is trained to reconstruct a spherical light field in cubemap projection (CMP) format, reconstructing a spherical light field (LF) from a single one of the omnidirectional images, where the spherical LF represents spatial and angular information of the scene, by reconstructing each of six cube faces

(

L

CMP

i

)

of cubes in the cubemap into an array of the cube faces

(

L

^

CMP

i

(

x

,

u

)

)

.

2 . The image processing method of claim 1 , further comprising, in a refinement network, removing border artifacts from the cubemap and minimizing a loss function to extract image features for both translation and zooming in the cubemap.

3 . The image processing method of claim 1 , in which the camera captures spherical images.

4 . The image processing method of claim 2 , further comprising:

converting each cubemap, from which the border artifacts have been removed, into the ERP format; and

applying the converted cubemap to a display system for viewing by a user.

5 . The image processing method of claim 1 , further comprising,

reconstructing a plurality of omnidirectional sub-aperture video frames for each omnidirectional image received from the camera, where each sub-aperture video frame corresponds to a respective perspective of the scene;

receiving from a moving display system information indicating a position and orientation of the moving display system;

rendering additional video frames from the sub-aperture of the captured scene from a current viewpoint of the moving display system according to its current orientation and position; and

causing the rendered additional video frames to be overlayed onto a default view of the scene displayed by the moving display system.

6 . The method of claim 5 , further comprising rendering at least one element and compositing each element onto each sub-aperture video frame.

7 . The method of claim 6 , further comprising:

calculating a respective depth in the scene at which each element is to be placed;

adjusting a size and angular orientation of each respective element according to its depth in the scene to be overlayed;

adjusting a perspective of each element when a viewpoint of the display system has changed; and

masking the rendered view of the element corresponding to depth and/or distance-dependent visual occlusion caused by other real objects existing in the scene and being displayed.

8 . An image processing system comprising:

a camera that captures omnidirectional spherical images of an imaged scene, each of which is transmitted in an equirectangular projection (ERP) and converted into a cubemap in a cubemap projection (CMP) format;

a computing platform including a reconstruction network that is trained to reconstruct a spherical light field in cubemap projection (CMP) format, and that reconstructs a spherical light field (LF) from a single one of the spherical images, where the spherical LF represents spatial and angular information of the scene, by reconstructing each of six cube faces

(

L

CMP

I

)

of cubes in the cube map into an array of the cube faces

(

L

CMP

I

(

x

,

u

)

)

.

9 . The image processing system of claim 8 , further comprising a refinement network that removes border artifacts from the cubemap and minimizes a loss function to extract image features for both translation and zooming in the cubemap.

10 . The image processing system of claim 9 , in which:

the refinement network further performs CMP-to-ERP format conversion of the cubemap; and

further comprising a display that receives and displays the converted cubemap for viewing by a user.