System for and method of generating user-selectable novel views on a viewing device

1. A method of generating user-selectable novel views of an event on a viewing device, the method comprising the steps of:

(a) reconstructing by a server system for each camera of a plurality of cameras image data into at least one foreground model for the respective camera and at least one environment model for the respective camera;

(b) joining by the server system the at least one foreground model for each camera to create a visual atlas of all foreground models;

(c) creating by the server system foreground mapping data for foreground image data in the visual atlas to a 3D coordinate in a 3D data representation;

(d) projecting by the server system environment image data of all cameras for each camera onto each respective environment model;

(e) creating by the server system environment mapping data for environment image data in each respective environment model to a 3D coordinate in a 3D data representation;

(f) saving by the server system for each camera the respective environment model as an environment image;

(g) compressing by the server system the foreground mapping data, the environment mapping data, and a depth map;

(h) inserting by the server system the visual atlas into each respective environment model as a new image frame in an image sequence by projecting background rasters on each respective environment model;

(i) compressing by the server system the visual atlas;

(j) compressing by the server system the respective environment images and the respective environment models;

(k) transmitting by the server system each compressed data in a sequence it was compressed;

(l) receiving by the viewing device all compressed data;

(m) uncompressing by the viewing device all compressed data;

(n) selecting by a user on the viewing device the novel view; and

(o) rendering by the viewing device the respective environment images onto the respective environment models for each novel view.

2. The method of claim 1 , further comprising before step (a), the steps of

imaging with the plurality of cameras the event in a plurality of image frames, the event comprising a foreground and an environment;

extracting by the server system image data from the plurality of image frames.

3. The method of claim 2 , further comprising the step of

saving the image data as the 3D data representation.

4. The method of claim 1 , further comprising after step (e) the step of

calculating by the server system a median model value for each respective environment model.

5. The method of claim 4 , further comprising after step (f) the step of

calculating by the server system a median image value for each respective environment image;

determining a difference between the median image value and the median model value.

6. The method of claim 1 , further comprising the step of

inserting by the server system a brightness image into the environment image.

7. The method of claim 6 , further comprising the step of

compressing by the server system the brightness image.

8. The method of claim 1 , wherein step (h) requires using an exponential matrix to decide the weight of a projection.

9. The method of claim 1 , wherein step (h) requires looping through each camera and referencing image data in the visual atlas.

10. An interactive-player system for generating user-selectable novel views of an event on a viewing device, the multi-view reconstruction system comprising:

a server system and a viewing device;

the server system

(a) reconstructing for each camera of a plurality of cameras image data into at least one foreground model for the respective camera and at least one environment model for the respective camera;

(b) joining the at least one foreground model for each camera to create a visual atlas of all foreground models;

(c) creating foreground mapping data for foreground image data in the visual atlas to a 3D coordinate in a 3D data representation;

(d) projecting environment image data of all cameras for each camera onto each respective environment model;

(e) creating environment mapping data for environment image data in each respective environment model to a 3D coordinate in a 3D data representation;

(f) saving for each camera the respective environment model as an environment image;

(g) compressing the foreground mapping data, the environment mapping data, and a depth map;

(h) inserting the visual atlas into each respective environment model as a new image frame in an image sequence by projecting background rasters on each respective environment model;

(i) compressing the visual atlas;

(j) compressing the respective environment images and the respective environment models;

(k) transmitting each compressed data in a sequence it was compressed;

the viewing device

receiving all compressed data;

uncompressing all compressed data;

selecting by a user on the viewing device the novel view; and

rendering by the viewing device the respective environment images onto the respective environment models for each novel view.

11. The system of claim 10 , wherein the viewing device comprises a path selection module for selecting a path to display a plurality of novel views.

12. The system of claim 10 , wherein the server side comprises a configuration engine for executing (a)-(k).

13. The system of claim 10 , further comprising

the plurality of cameras for imaging the event in a plurality of image frames, the event comprising a foreground and an environment;

an FES module for separating the foreground from the environment in the image frames.

14. The system of claim 10 , further comprising a plurality of viewing devices and a social messaging system for communicating among a plurality of users of viewing devices.