Apparatus and method for augmented reality

1. A method of incorporating an augmented reality video image into a virtual environment, comprising:

detecting, by a first image analyser, a state of a real-world control object captured in a video image, the real-world control object being a physical object configured to be pointed in a certain direction by a user;

augmenting, by one or more image processors, the video image responsive to the state of the real-world control object;

calculating, by one or more trajectory processors, trajectory parameters for an action with respect to a local co-ordinate system of the augmented video image, responsive to the state of the real-world control object, the trajectory parameters for the action relating to a pointing direction of the real-world control object;

embedding, by the one or more image processors, the augmented video image within a 3D virtual environment;

calculating, by the one or more trajectory processors, the trajectory parameters for the action with respect to a local co-ordinate system of the 3D virtual environment; and

rendering, by the one or more image processors, a graphical effect corresponding to the action within the 3D virtual environment, the rendered graphical effect being adapted for presentation on a display device to the user.

2. A method according to claim 1 , comprising the steps of:

calculating, according to the trajectory parameters, a point of intersection with an image plane of the video image; and

calculating a corresponding point within the 3D virtual environment in which the augmented video image is embedded;

and in which:

the step of augmenting the video image responsive to the state of the real-world control object comprises augmenting the video image with the graphical effect corresponding to the action; and

the step of rendering the graphical effect corresponding to the action within the 3D virtual environment comprises rendering the graphical effect with a start-point at the calculated corresponding point within the 3D virtual environment.

3. A method according to claim 1 , comprising the steps of:

calculating a position and orientation of the real-world control object with respect to the local co-ordinate system of the video image; and

mapping the calculated position and orientation of the real-world control object to the local co-ordinate system of the 3D virtual environment, responsive to a position and orientation of the augmented video image embedded within the 3D virtual environment;

and in which:

the step of rendering the graphical effect corresponding to the action within the 3D virtual environment comprises rendering the graphical effect with a start-point at the start of the calculated trajectory.

4. A method according to claim 3 , in which the step of rendering the graphical effect corresponding to the action within the 3D virtual environment comprises modifying a depth buffer to allow display of the graphical effect when the augmented video image is positioned between the start of the calculated trajectory and the rendering viewpoint.

5. A method according to claim 1 , in which the calculated trajectory is one or more selected from the list consisting of:

i. a linear vector;

ii. a parametrically defined non-linear path; and

iii. a procedurally defined non-linear path.

6. A method according to claim 1 , in which the step of embedding the augmented video image within the 3D virtual environment comprises scaling the augmented video image responsive to the size of the control object within the captured video image.

7. A method according to claim 1 in which the step of embedding the augmented video image within the 3D virtual environment comprises applying a graphical effect to the augmented video image that obscures the edges of the augmented video image.

8. A method according to claim 1 , comprising the step of modifying a colour balance of the video image in response to one or more selected from the list consisting of:

i. a colour temperature of a virtual object used in the augmentation of the video image; and

ii. a colour temperature of the 3D virtual environment.

9. A method according to claim 1 , comprising the step of modifying a brightness of the video image in response to one or more selected from the list consisting of:

i. a predefined brightness of a virtual object used in the augmentation of the video image; and

ii. a brightness value for the 3D virtual environment.

10. A non-transitory computer program product comprising computer program instructions that when performed by one or more processors implement a method comprising the steps of:

detecting a state of a real-world control object captured in a video image, the real-world control object being a physical object configured to be pointed in a certain direction by a user;

augmenting the video image responsive to the state of the real-world control object;

calculating trajectory parameters for an action with respect to a local co-ordinate system of the augmented video image, responsive to the state of the real-world control object, the trajectory parameters for the action relating to a pointing direction of the real-world control object;

embedding the augmented video image within a 3D virtual environment;

calculating the trajectory parameters for the action with respect to a local co-ordinate system of the 3D virtual environment; and

rendering a graphical effect corresponding to the action within the 3D virtual environment, the rendered graphical effect being adapted for presentation on a display device to the user.

11. An entertainment device, comprising

an input configured to receive a captured video image depicting a real-world control object;

a first image analyser configured to detect a state of the real-world control object in the video image, the real-world control object being a physical object configured to be pointed in a certain direction by a user;

one or more image processors configured to augment the video image responsive to the state of the control object;

one or more trajectory processors configured to calculate trajectory parameters for an action with respect to a local co-ordinate system of the augmented video image, responsive to the state of the control object, the trajectory parameters for the action relating to a pointing direction of the real-world control object;

the one or more image processors configured to embed the augmented video image within a 3D virtual environment;

the one or more trajectory processors configured to calculate the trajectory parameters for the action with respect to a local co-ordinate system of the 3D virtual environment; and

the one or more image processors configured to render a graphical effect corresponding to the action within the 3D virtual environment, the rendered graphical effect being adapted for presentation on a display device to the user.

12. An entertainment device according to claim 11 , in which:

the one or more trajectory processors are configured to calculate, according to the trajectory parameters, a point of intersection with an image plane of the video image; and

the one or more trajectory processors are configured to calculate a corresponding point within the 3D virtual environment in which the augmented video image is embedded;

the one or more image processors are configured to augment the video image with the graphical effect corresponding to the action; and

the one or more image processors are configured to render the graphical effect with a start-point at the calculated corresponding point within the 3D virtual environment.

13. An entertainment device according to claim 11 , in which:

the one or more trajectory processors are configured to calculate a position and orientation of the real-world control object with respect to the local co-ordinate system of the video image; and

the one or more trajectory processors are configured to map the calculated position and orientation of the real-world control object to the local co-ordinate system of the 3D virtual environment, responsive to a position and orientation of the augmented video image embedded within the 3D virtual environment;

and in which:

the one or more image processors are configured to render the graphical effect with a start-point at the start of the calculated trajectory.

14. An entertainment device according to claim 13 , comprising a depth buffer arranged to allow display of the graphical effect when the augmented video image is positioned between the start of the calculated trajectory and the rendering viewpoint.

15. An entertainment device according to claim 11 , in which the one or more image processors are configured to scale the augmented video image responsive to the size of the control object within the captured video image when embedding the augmented video image within the 3D virtual environment.