4D camera tracking and optical stabilization

1. A method for processing a light-field video stream, the method comprising:

at a processor, receiving a light-field video stream generated by a light-field camera;

at the processor, selecting a plurality of target pixels in a plurality of key frames comprising at least a first frame and a second frame of the light-field video stream, wherein selecting the plurality of target pixels comprises selecting as target pixels those pixels having at least one of a predetermined color characteristic or a predetermining intensity characteristic and are selected from planar regions of static, textured objects appearing in the key frames of the light-field video stream as the target pixels, and wherein selecting the plurality of target pixels further comprises:

generating a list of a plurality of targets appearing in at least one of the key frames;

generating a plane model of each of the targets for each of the key frames; and

generating a mask for each of the targets for each of the key frames, indicating one or more target pixels within each of the targets;

at the processor, using the target pixels to generate, in three dimensions, a camera pathway indicative of motion of the light-field camera during generation of the light-field video stream; and

at the processor, using the generated camera pathway to process the light-field video stream.

2. The method of claim 1 , wherein selecting the plurality of target pixels further comprises:

performing superpixel segmentation of each of the key frames to identify superpixels;

calculating a motion error map for each of the key frames;

using the superpixels and motion error maps to assess texture and motion error for each of the superpixels of each of the key frames to identify a plurality of the superpixels as candidate targets;

fitting a plane to each of the candidate targets for each of the key frames; and

selecting the targets from among the candidate targets.

3. The method of claim 1 , wherein using the generated camera pathway to process the light-field video stream comprises:

adjusting the camera pathway to generate an adjusted camera pathway; and

projecting the light-field video stream to a viewpoint defined by the adjusted camera pathway to generate a projected video stream;

wherein the method further comprises, at an output device, outputting the projected video stream.

4. A method for processing a light-field video stream, the method comprising:

at a processor, receiving a light-field video stream generated by a light-field camera;

at the processor, selecting a plurality of target pixels in a plurality of key frames comprising at least a first frame and a second frame of the light-field video stream, wherein selecting the plurality of target pixels comprises selecting as target pixels those pixels having at least one of a predetermined color characteristic or a predetermining intensity characteristic;

at the processor, using the target pixels to generate, in three dimensions, a camera pathway indicative of motion of the light-field camera during generation of the light-field video stream;

at the processor, using the generated camera pathway to process the light-field video stream; and

wherein using the target pixels to generate the camera pathway comprises:

receiving at least one selection from the group consisting of:

camera-intrinsic parameters obtained from calibration of the light-field camera;

light-field optics parameters pertinent to one or more light-field optical elements of the light-field camera; and

camera motion, generated by a sensor operating contemporaneously with capture of the light-field video stream, indicative of motion of the light-field camera during at least a segment of the light-field video stream; and

using the selection to generate the camera pathway.

5. The method of claim 4 , wherein using the target pixels to generate the camera pathway comprises:

generating the camera pathway with six degrees of freedom for an entirety of the light-field video stream; and

generating a 3D mapping of the target pixels.

6. A method for processing a light-field video stream, the method comprising:

at a processor, receiving a light-field video stream generated by a light-field camera;

at the processor, selecting a plurality of target pixels in a plurality of key frames comprising at least a first frame and a second frame of the light-field video stream;

at the processor, using the target pixels to generate, in three dimensions, a camera pathway indicative of motion of the light-field camera during generation of the light-field video stream, wherein using the target pixels to generate the camera pathway comprises:

dividing the light-field video stream into a plurality of sequences, each of which begins with one of the key frames;

for each sequence:

tracking a position and/or orientation of each of the target pixels in each frame;

comparing the position and/or orientation of each of the target pixels between frames of the sequence to obtain a portion of the camera pathway for that sequence; and

obtaining the position and/or orientation of each of the target pixels for the key frame for the next sequence;

generating the camera pathway with six degrees of freedom for an entirety of the light-field video stream; and

generating a 3D mapping of the target pixels; and

at the processor, using the generated camera pathway to process the light-field video stream.

7. The method of claim 6 , wherein using the generated camera pathway to process the light-field video stream comprises:

adjusting the camera pathway to generate an adjusted camera pathway; and

projecting the light-field video stream to a viewpoint defined by the adjusted camera pathway to generate a projected video stream;

wherein the method further comprises, at an output device, outputting the projected video stream.

8. The method of claim 7 , wherein adjusting the camera pathway to generate the adjusted camera pathway comprises causing the adjusted camera pathway to be more stable than the camera pathway.

9. The method of claim 7 , wherein adjusting the camera pathway to generate the adjusted camera pathway further comprises adjusting U, V coordinates within each of a plurality of frames of the light-field video stream to cause frame-to-frame motion to be relatively smooth and contiguous.

10. A non-transitory computer-readable medium for processing a light-field video stream, comprising instructions stored thereon, that when executed by one or more processors, perform the steps of:

receiving a light-field video stream generated by a light-field camera;

selecting a plurality of target pixels in a plurality of key frames comprising at least a first frame and a second frame of the light-field video stream, wherein selecting the plurality of target pixels comprises:

selecting pixels with predetermined color and/or intensity characteristics by selecting pixels from planar regions of static, textured objects appearing in the key frames of the light-field video stream as the target pixels by:

generating a list of a plurality of targets appearing in at least one of the key frames;

generating a plane model of each of the targets for each of the key frames; and

generating a mask for each of the targets for each of the key frames, indicating one or more target pixels within each of the targets;

using the target pixels to generate, in three dimensions, a camera pathway indicative of motion of the light-field camera during generation of the light-field video stream; and

using the generated camera pathway to process the light-field video stream.

11. The non-transitory computer-readable medium of claim 10 , wherein selecting the plurality of target pixels further comprises:

performing superpixel segmentation of each of the key frames to identify superpixels;

calculating a motion error map for each of the key frames;

using the superpixels and motion error maps to assess texture and motion error for each of the superpixels of each of the key frames to identify a plurality of the superpixels as candidate targets;

fitting a plane to each of the candidate targets for each of the key frames; and

selecting the targets from among the candidate targets.

12. The non-transitory computer-readable medium of claim 10 , wherein using the generated camera pathway to process the light-field video stream comprises:

adjusting the camera pathway to generate an adjusted camera pathway such that the adjusted camera pathway is more stable than the camera pathway;

projecting the light-field video stream to a viewpoint defined by the adjusted camera pathway to generate a projected video stream;

wherein the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by one or more processors, cause an output device to output the projected video stream.

13. A system for processing a light-field video stream, the system comprising:

a processor configured to:

receive a light-field video stream generated by a light-field camera;

select a plurality of target pixels in a plurality of key frames comprising at least a first frame and a second frame of the light-field video stream by:

selecting pixels from planar regions of static, textured objects appearing in the key frames of the light-field video stream as the target pixels;

generating a list of a plurality of targets appearing in at least one of the key frames;

generating a plane model of each of the targets for each of the key frames; and

generating a mask for each of the targets for each of the key frames, indicating one or more target pixels within each of the targets;

use the target pixels to generate, in three dimensions, a camera pathway indicative of motion of the light-field camera during generation of the light-field video stream; and

use the generated camera pathway to process the light-field video stream; and

an output device configured to output the light-field video stream.

14. The system of claim 13 , wherein the processor is further configured to select the plurality of target pixels by:

performing superpixel segmentation of each of the key frames to identify superpixels;

calculating a motion error map for each of the key frames;

using the superpixels and motion error maps to assess texture and motion error for each of the superpixels of each of the key frames to identify a plurality of the superpixels as candidate targets;

fitting a plane to each of the candidate targets for each of the key frames; and

selecting the targets from among the candidate targets.

15. The system of claim 13 , wherein the processor is further configured to use the target pixels to generate the camera pathway by:

receiving at least one selection from the group consisting of:

camera-intrinsic parameters obtained from calibration of the light-field camera;

light-field optics parameters pertinent to one or more light-field optical elements of the light-field camera; and

camera motion, generated by a sensor operating contemporaneously with capture of the light-field video stream, indicative of motion of the light-field camera during at least a segment of the light-field video stream; and

using the selection to generate the camera pathway.

16. The system of claim 13 , wherein the processor is further configured to use the generated camera pathway to process the light-field video stream by:

adjusting the camera pathway to generate an adjusted camera pathway such that the adjusted camera pathway is more stable than the camera pathway; and

projecting the light-field video stream to a viewpoint defined by the adjusted camera pathway to generate a projected video stream;

and wherein the output device is further configured to output the projected video stream.

17. A system for processing a light-field video stream, the system comprising:

a processor configured to:

receive a light-field video stream generated by a light-field camera;

select a plurality of target pixels in a plurality of key frames comprising at least a first frame and a second frame of the light-field video stream;

use the target pixels to generate, in three dimensions, a camera pathway with six degrees of freedom and indicative of motion of the light-field camera during generation of the light-field video stream by:

dividing the light-field video stream into a plurality of sequences, each of which begins with one of the key frames; and

for each sequence:

tracking a position and/or orientation of each of the target pixels in each frame;

comparing the position and/or orientation of each of the target pixels between frames of the sequence to obtain a portion of the camera pathway for that sequence; and

obtaining the position and/or orientation of each of the target pixels for the key frame for the next sequence; and

generating a 3D mapping of the target pixels;

use the generated camera pathway to process the light-field video stream; and

an output device configured to output the light-field video stream.

18. The system of claim 17 , wherein the processor is further configured to use the generated camera pathway to process the light-field video stream by:

adjusting the camera pathway to generate an adjusted camera pathway such that the adjusted camera pathway is more stable than the camera pathway; and

projecting the light-field video stream to a viewpoint defined by the adjusted camera pathway to generate a projected video stream;

and wherein the output device is further configured to output the projected video stream.