System and method of multi-view reconstruction with user-selectable novel views

1. A system for multi-view reconstruction of a photo-realistic rendering of an event, the system comprising:

a plurality of cameras for imaging the event with a plurality of image frames;

a controller comprising one or more processors and a computer readable storage medium including instructions that, when executed by the one or more processors, cause the controller to implement:

a CEM module to model an environment from image data of the image frames;

an FES module to segment a foreground from the environment from image data of the image frames and constructing a 3D data representation; and

a configuration engine comprising a path selection module, the configuration engine to configure and render the photo-realistic rendering along a path selected by a user using the path selection module, the path comprising at least one novel view image;

wherein the photo-realistic rendering requires less than 10% discrepancy between output pixel raster values of the novel view image and the image frames imaged by the cameras, the discrepancy based at least on a change in position or orientation of the novel view image and the image frames imaged by the cameras, the discrepancy calculated using a pixel-wise RMS comparison in RGB color space between a plurality of sections of the novel view image and a corresponding plurality of sections of the image frames, wherein RMS=√{square root over (|R m −R g | 2 +|G m −G g | 2 +|B m −B g | 2 )}, wherein R, G, B represent RGB values, X m denotes an image pixel value from the novel view image and X g represents an image pixel value from the image frames imaged by the cameras.

2. The system of claim 1 , wherein the FES module comprises a background state machine to analyze if a pixel located in a same position in the plurality of image frames has changed color according to a predetermined level in the plurality of image frames.

3. The system of claim 1 , wherein the FES module comprises a background state machine to analyze if a pixel detected has changed color according to a predetermined level in the plurality of image frames.

4. The system of claim 1 , wherein the FES module calculates the mean color value using the RGB color space.

5. The system of claim 1 , further comprising a shader for determining the color information of a pixel based on the image frames which comprise the pixel.

6. The system of claim 1 , further comprising an imaging server having a plurality of memories, wherein one memory stores image data from the image frames and another memory is used by the FES module for segmentation.

7. The system of claim 1 , further comprising a foreground shading module for matching colors within the at least one novel view image.

8. The system of claim 1 , further comprising

an imaging server associated with at least one camera of the plurality of cameras; and

a plurality of communications links between at least one camera and the imaging server.

9. The system of claim 8 , wherein the imaging server is in electronic communication with the controller.

10. A method of reconstructing of a photo-realistic rendering of an event with user-selectable novel views, the method comprising:

(a) imaging the event with a plurality of cameras for producing a plurality of image frames, the event comprising a foreground and an environment;

(b) modeling, by a CEM module, the environment from image data of the image frames;

(c) segmenting, by an FES module, the foreground from the environment from image data of the image frames and constructing a 3D data representation;

(d) configuring and rendering, by a configuration engine, a replay of the event;

(e) receiving, by a viewing device, the replay for display to a viewer; and

(f) selecting, by a user using a path selection module on the viewing device, a viewing path comprising at least one novel view image;

wherein the photo-realistic rendering requires less than 10% discrepancy between output pixel raster values of the novel view image and the image frames imaged by the cameras, the discrepancy based at least on a change in position or orientation of the novel view image and the image frames imaged by the cameras, the discrepancy calculated using a pixel-wise RMS comparison in RGB color space between a plurality of sections of the novel view image and a corresponding plurality of sections of the image frames, wherein RMS=√{square root over (|R m −R g | 2 +|G m −G g | 2 +|B m −B g | 2 )}, wherein R, G, B represent RGB values, X m denotes an image pixel value from the novel view image and X g represents an image pixel value from the image frames imaged by the cameras.

11. The method of claim 10 , further comprising a step of executing a sub-space division to generate a run-length encoding database.

12. The method of claim 10 , further comprising a step of encoding, the configuration engine, a transmission to the viewing device.

13. The method of claim 12 , wherein steps (b) and (c) are executed on a memory of an imaging server associated with at least one camera of the plurality of the cameras.

14. The method of claim 10 , wherein step (c) is performed by analyzing if a pixel that has been detected in the plurality of image frames has changed color according to a predetermined level in the plurality of image frames.

15. The method of claim 10 , further comprising a step of performing a bundle adjustment of the plurality of cameras to determine a location of each camera.

16. The method of claim 10 , further comprising a step of calibrating, by the CEM module, image data of the image frames to develop a coherent 3D data representation.

17. A non-transitory computer readable storage medium including instructions stored thereon which, when executed by a processor, cause the processor to:

image the event with a plurality of cameras for producing a plurality of image frames, the event comprising a foreground and an environment;

model, by a CEM module, the environment from image data of the image frames;

segment, by an FES module, the foreground from the environment from image data of the image frames and constructing a 3D data representation;

configure and render, by a configuration engine, a replay of the event;

receive, by a viewing device, the replay for display to a viewer; and

select, by a user using a path selection module on the viewing device, a viewing path comprising at least one novel view image;

wherein the photo-realistic rendering comprises requires less than 10% discrepancy between output pixel raster values of the novel view image and the image frames imaged by the cameras, the discrepancy based at least on a change in position or orientation of the novel view image and the image frames imaged by the cameras, the discrepancy calculated using a pixel-wise RMS comparison in RGB color space between a plurality of sections of the novel view image and a corresponding plurality of sections of the image frames, wherein RMS=√{square root over (|R m −R g | 2 +|G m −G g | 2 +|B m −B g | 2 )}, wherein R, G, B represent RGB values, X m denotes an image pixel value from the novel view image and X g represents an image pixel value from the image frames imaged by the cameras.

18. The non-transitory computer readable storage medium of claim 17 , further comprising a step of executing a sub-space division to generate a run-length encoding database.

19. The non-transitory computer readable storage medium of claim 17 , further comprising a step of encoding, the configuration engine, a transmission to the viewing device.

20. The non-transitory computer readable storage medium of claim 17 , wherein the instructions to segment, by an FES module, the foreground from the environment from image data of the image frames and constructing a 3D data representation, further cause the processor to:

analyze if a pixel that has been detected in the plurality of image frames has changed color according to a predetermined level in the plurality of image frames.