Computer-generated image processing including volumetric scene reconstruction

1. A computer-implemented method of processing images from a main imaging device using capture device inputs from capture devices, the method comprising acts of:

obtaining a main video sequence using a main camera capturing color images of objects visible in a live action scene including capturing a primary object in color visible in the live action scene, wherein the main video sequence comprises a main pixel array including pixel color values from the color images captured by the main camera at a main camera view of the live action scene;

obtaining, concurrently with obtaining the main video sequence, additional image data of the live action scene using at least one ancillary device capable of capturing color from the live action scene, wherein the at least one ancillary device is positioned to capture visual information from an obscured object that is obscured, at least in part, from the main camera view by the primary object;

mapping the additional image data into corresponding volumetric data according to main camera rays from the main camera and the additional image data captured by the ancillary devices;

determining one or more portions of the main pixel array associated with the primary object to be replaced;

digitally removing the primary object in a synthetic image by replacing the pixel color values of the primary object captured by the main camera, with replacement pixel color values associated with the obscured object, wherein replacement pixel color values of the synthetic pixel array comprise pixel color values of pixels of portions of the obscured object captured by the at least one ancillary device and pixel color values of pixels from the volumetric data; and

merging computer-generated content with the synthetic image to generate an output image.

2. The method of claim 1 , wherein the ancillary device is positioned on a set at a differing angle from the main camera and wherein mapping the additional image data into volumetric data comprises performing geometric transformations to account for the differing angle.

3. The method of claim 1 , wherein the synthetic image comprises a plurality of discontinuous replacement portions.

4. The method of claim 1 , further comprising obtaining user input for determining which objects in the main video sequence to replace using the volumetric data.

5. The method of claim 1 wherein a pixel value includes two or more channels.

6. The method of claim 5 , wherein a channel includes one or more of a color value, an alpha value, a pan-chromatic definition, or a shadow map value.

7. The method of claim 1 , where the one or more portions of the main pixel array include a two-dimensional portion.

8. The method of claim 1 , where the one or more portions of the main pixel array include a three-dimensional portion.

9. The method of claim 1 , where the main camera includes a digital camera.

10. The method of claim 1 , wherein two or more main cameras are used.

11. The method of claim 1 , wherein two stereo main cameras are used.

12. The method of claim 1 , where the ancillary device includes one or more of an optical camera, an infrared camera, a Lidar sensor, a sonar sensor, or a dot flood system.

13. The method of claim 1 , where a second object is identified at least in part by tracing a ray from a particular capture device along a view line toward the obscured object.

14. An apparatus using a digital processor to perform the acts of claim 1 .

15. One or more processor-readable tangible media including instructions executable by one or more processors to perform the acts of claim 1 .

16. The method of claim 1 , wherein the ancillary device is a motion capture device and wherein the primary object wears one or more motion capture fiducials used to produce the computer-generated content based on the removed primary object.