Automatic composition of composite images or video with stereo foreground objects

1. A method for generating a composite output image from an input video having a sequence of frames, the method comprising:

receiving a sequence of stereo video frames depicting a foreground object;

for selected frames in the sequence of stereo video frames, performing by a processing device, foreground/background segmentations to extract respective stereo foreground object images each comprising a representation of the foreground object with background pixels subtracted;

storing the respective stereo foreground object images to a foreground object list, each of the stereo foreground object images having left and right images with a disparity between them;

transforming the stereo foreground object images to adjust the respective disparities between the respective left and right images based on a change between a convergence depth for the respective selected frames and a convergence depth for a current frame, the transforming of the stereo foreground object images comprising:

decreasing the disparity for a given stereo foreground object image, in response to the convergence depth for the current frame being further away than a convergence depth for a selected frame corresponding to the given stereo foreground object; or

increasing the disparity for the given stereo foreground object image, in response to the convergence depth for the current frame being closer than the convergence depth for the selected frame corresponding to the given stereo foreground object; and

overlaying the transformed stereo foreground object images onto the current frame to generate a composite output image.

2. The method of claim 1 , further comprising setting the convergence depth for the current frame based on a depth of a current position of the foreground object in the current frame relative to a current camera position when the current frame is captured.

3. The method of claim 1 , wherein the transforming of the stereo foreground object images further comprises scaling the representation of the foreground object in the stereo foreground object image based on camera motion representing a change in position of a camera between capturing the selected frame and capturing the current frame.

4. The method of claim 3 , wherein the transforming of the stereo foreground object images further comprises:

enlarging the representation of the foreground object in the stereo foreground object image in response to the camera being closer to a position of the foreground object when the selected frame was captured; and otherwise

reducing in size the representation of the foreground object in the foreground object image in response to the camera being further from the position of the foreground object when the selected frame was captured.

5. The method of claim 1 , wherein the transforming of the stereo foreground object images further comprises translating a location of the representation of the stereo foreground object in the foreground object image based on a camera motion representing a change in planar position of a camera between capturing the selected frame and capturing the current frame.

6. The method of claim 5 , wherein the transforming of the stereo foreground object images further comprises translating the location according to a motion equal and opposite to the camera motion.

7. The method of claim 1 , wherein the transforming of the stereo foreground object images further comprises applying a rotation to the stereo foreground object image based on camera motion representing a change in rotational position of a camera about the foreground object between capturing the selected frame and capturing the current frame.

8. The method of claim 1 , further comprising:

detecting an overlaid stereo foreground object image exceeding an overlap threshold of at least one other overlaid stereo foreground object image; and

applying a partial transparency to the detected overlaid stereo foreground object image exceeding the overlap threshold.

9. The method of claim 1 , wherein the performing of the foreground/background segmentations comprises applying an adaptive Gaussian Mixture Model as a predictive model.

10. A non-transitory computer-readable storage medium configured to store instructions for generating a composite output image from an input video having a sequence of frames, the instructions being configured to cause, when executed by a processor apparatus, the processor apparatus to perform steps comprising:

receiving a sequence of stereo video frames depicting a foreground object;

for selected frames in the sequence of stereo video frames, performing foreground/background segmentations to extract respective stereo foreground object images each comprising a representation of the foreground object with background pixels subtracted;

storing the respective stereo foreground object images to a foreground object list, each of the stereo foreground object images having left and right images with a disparity between them;

transforming the stereo foreground object images to adjust the respective disparities between the respective left and right images based on a change between a convergence depth for the respective selected frames and a convergence depth for a current frame; and

overlaying the transformed stereo foreground object images onto the current frame to generate a composite output image;

wherein the transforming of the stereo foreground object images comprises:

decreasing the disparity for a given stereo foreground object image, in response to the convergence depth for the current frame being further away than a convergence depth for a selected frame corresponding to the given stereo foreground object; and

increasing the disparity for the given stereo foreground object image, in response to the convergence depth for the current frame being closer than the convergence depth for the selected frame corresponding to the given stereo foreground object.

11. The non-transitory computer-readable storage medium of claim 10 , wherein the instructions, when executed by the processor apparatus, are further configured to cause the processor apparatus to perform a step of:

setting the convergence depth for the current frame based on a depth of a current position of the foreground object in the current frame relative to a current camera position when the current frame is captured.

12. The non-transitory computer-readable storage medium of claim 10 , wherein the transforming of the stereo foreground object images further comprises:

scaling the representation of the stereo foreground object in the foreground object image based on camera motion representing a change in position of a camera between capturing the selected frame and capturing the current frame.

13. The non-transitory computer-readable storage medium of claim 12 , wherein the transforming of the stereo foreground object images further comprises:

enlarging the representation of the stereo foreground object in the stereo foreground object image in response to the camera being closer to a position of the foreground object when the selected frame was captured; or otherwise

reducing in size the representation of the foreground object in the foreground object image in response to the camera being further from the position of the foreground object when the selected frame was captured.

14. The non-transitory computer-readable storage medium of claim 10 , wherein the transforming of the stereo foreground object images further comprises:

translating a location of the representation of the stereo foreground object in the foreground object image based on a camera motion representing a change in planar position of a camera between capturing the selected frame and capturing the current frame.

15. The non-transitory computer-readable storage medium of claim 14 , wherein the transforming of the stereo foreground object images further comprises translating the location according to a motion equal and opposite to the camera motion.

16. The non-transitory computer-readable storage medium of claim 10 , wherein the transforming of the stereo foreground object images further comprises:

applying a rotation to the stereo foreground object image based on camera motion representing a change in rotational position of a camera about the foreground object between capturing the selected frame and capturing the current frame.

17. The non-transitory computer-readable storage medium of claim 10 , wherein the instructions, when executed by the processor apparatus, further cause the processor apparatus to perform steps including:

detecting an overlaid stereo foreground object image exceeding an overlap threshold of at least one other overlaid stereo foreground object image; and

applying a partial transparency to the detected overlaid stereo foreground object image exceeding the overlap threshold.

18. The non-transitory computer-readable storage medium of claim 10 , wherein the performing of the foreground/background segmentations comprises applying an adaptive Gaussian Mixture Model as a predictive model.

19. An apparatus configured to generate a composite output image from an input video having a sequence of frames, the apparatus comprising:

a processor apparatus;

a non-transitory computer-readable storage medium having a computer application, the computer application comprising a plurality of instructions configured to, when executed by the processor apparatus, cause the apparatus to:

receive a sequence of stereo video frames depicting a foreground object;

for selected frames in the sequence of stereo video frames, perform foreground/background segmentations to extract respective stereo foreground object images each comprising a representation of the foreground object with background pixels subtracted;

store the respective stereo foreground object images to a foreground object list, each of the stereo foreground object images having left and right images with a disparity between them;

transform the stereo foreground object images to adjust the respective disparities between the respective left and right images based on a change between a convergence depth for the respective selected frames and a convergence depth for a current frame; and

overlay the transformed stereo foreground object images onto the current frame to generate a composite output image;

wherein, during the transformation of the stereo foreground object images, the plurality of instructions are further configured to, when executed by the processor apparatus, cause the apparatus to:

decrease the disparity for a given stereo foreground object image, in response to the convergence depth for the current frame being further away than a convergence depth for a selected frame corresponding to the given stereo foreground object; and

increase the disparity for the given stereo foreground object image, in response to the convergence depth for the current frame being closer than the convergence depth for the selected frame corresponding to the given stereo foreground object.

20. The apparatus of claim 19 , wherein the transformation of the stereo foreground object images comprises at least one of:

scaling of the representation of the foreground object in the stereo foreground object image based on camera motion representing a change in position of a camera between capturing the selected frame and capturing the current frame;

translation of a location of the representation of the stereo foreground object in the foreground object image based on a camera motion representing a change in planar position of a camera between capturing the selected frame and capturing the current frame; and

application of a rotation to the stereo foreground object image based on camera motion representing a change in rotational position of a camera about the foreground object between capturing the selected frame and capturing the current frame.