Game engine responsive to motion-capture data for mixed-reality environments
An immersive content presentation system can capture the motion or position of a performer in a real-world environment. A game engine can be modified to receive the position or motion of the performer and identify predetermined gestures or positions that can be used to trigger actions in a 3-D virtual environment, such as generating a digital effect, transitioning virtual assets through an animation graph, adding new objects, and so forth. The use of the 3-D environment can be rendered and composited views can be generated. Information for constructing the composited views can be streamed to numerous display devices in many different physical locations using a customized communication protocol. Multiple real-world performers can interact with virtual objects through the game engine in a shared mixed-reality experience.
1. A method comprising:
causing, by a game engine, a virtual object in a 3-D virtual environment to be in a first animation state in an animation graph;
receiving, by a plug-in of the game engine, one or more 3-D frames from a motion-capture camera system, wherein the plug-in is configured to:
receive 3-D frames from the motion-capture camera system as an input;
identify a motion or position of a performer in real time in a first real-world environment represented in the one or more 3-D frames; and
provide the motion or position of a performer as an input to the game engine;
maintaining a library of predefined motions or positions, wherein each of the predefined motions or positions includes a corresponding virtual asset or action;
identifying, by the game engine, the motion or position as one of the predefined motions or positions in the library; and
causing, by the game engine, the virtual object in the 3-D virtual environment to transition to a second animation state in the animation graph based on the corresponding action in the library, wherein the transition is triggered by identifying the motion or position as the one of the predefined motions or positions in the library.
2. The method of claim 1 , wherein the performer in the first real-world environment comprises a motion-capture suit that is recorded with a plurality of motion-capture cameras in a motion-capture system.
3. The method of claim 2 , wherein the motion or position of the real-time performer in the first real-world environment is represented by a sequence of motion-capture frame poses in a 3-D space recorded by the motion-capture cameras.
4. The method of claim 3 , wherein a motion-capture frame in the sequence of motion-capture frame poses comprises a plurality of vertices corresponding to visible markers on the performer in the first real-world environment.
5. The method of claim 4 , wherein the motion or position of the performer is further represented by motion vectors that are calculated for the plurality of vertices between motion-capture frame poses in the sequence of motion-capture frame poses.
6. The method of claim 1 , further comprising:
receiving, by the game engine, a second motion or position of the performer in real time in the first real-world environment;
identifying, by the game engine, the second motion or position as a second one of the predefined motions or positions in the library; and
causing, by the game engine, a virtual asset corresponding to the second one of the predefined motions or positions in the library to be added to the 3-D virtual environment, wherein adding the virtual asset to the 3-D virtual environment is triggered by identifying the second motion or position as the second one of the predefined motions or positions in the library.
7. A system comprising:
one or more processors; and
one or more memory devices comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
causing, by a game engine, a virtual object in a 3-D virtual environment to be in a first animation state in an animation graph;
receiving, by a plug-in of the game engine, one or more 3-D frames from a motion-capture camera system, wherein the plug-in is configured to:
receive 3-D frames from the motion-capture camera system as an input;
identify a motion or position of a performer in real time in a first real-world environment represented in the one or more 3-D frames; and
provide the motion or position of a performer as an input to the game engine;
maintaining a library of predefined motions or positions, wherein each of the predefined motions or positions includes a corresponding virtual asset or action;
identifying, by the game engine, the motion or position as one of the predefined motions or positions in the library; and
causing, by the game engine, the virtual object in the 3-D virtual environment to transition to a second animation state in the animation graph based on the corresponding action in the library, wherein the transition is triggered by identifying the motion or position as the one of the predefined motions or positions in the library.
8. The system of claim 7 , wherein the predefined motion or position comprises a hand or arm movement by the performer in the first real-world environment.
9. The system of claim 7 , wherein the predefined motion or position comprises a body posture of the performer in the first real-world environment.
10. The system of claim 7 , wherein the predefined motion or position comprises a fighting movement by the performer in the first real-world environment.
11. The system of claim 7 , wherein the predefined motion or position comprises a repeated movement by the performer in the first real-world environment.
12. The system e of claim 7 , wherein identifying the motion or position as the predefined motion or position comprises:
determining whether the motion or position was similar to the predefined motion or position within a threshold amount.
13. The system of claim 7 , wherein identifying the motion or position as the predefined motion or position determines how the first animation state is blended into the second animation state.
14. A non-transitory, computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
causing, by a game engine, a virtual object in a 3-D virtual environment to be in a first animation state in an animation graph;
receiving, by a plug-in of the game engine, one or more 3-D frames from a motion-capture camera system, wherein the plug-in is configured to:
receive 3-D frames from the motion-capture camera system as an input;
identify a motion or position of a performer in real time in a first real-world environment represented in the one or more 3-D frames; and
provide the motion or position of a performer as an input to the game engine;
maintaining a library of predefined motions or positions, wherein each of the predefined motions or positions includes a corresponding virtual asset or action;
identifying, by the game engine, the motion or position as one of the predefined motions or positions in the library; and
causing, by the game engine, the virtual object in the 3-D virtual environment to transition to a second animation state in the animation graph based on the corresponding action in the library, wherein the transition is triggered by identifying the motion or position as the one of the predefined motions or positions in the library.
15. The non-transitory, computer-readable medium of claim 14 , wherein:
the motion or position of the performer in the first real-world environment was performed during a previous training session; and
the game engine stored the motion or position of the performer as the predefined motion or position resulting from the previous training session.
16. The non-transitory, computer-readable medium of claim 14 , wherein the virtual object in the 3-D virtual environment comprises a digital effect that is not visible or active during the first animation state and is visible or active during the second animation state.
17. The non-transitory, computer-readable medium of claim 14 , wherein the virtual object in the 3-D virtual environment comprises a CGI character.
18. The non-transitory, computer-readable medium of claim 14 , wherein the game engine is configured to output one or more 3-D transforms that may be applied to the virtual object in the 3-D virtual environment.
19. The non-transitory, computer-readable medium of claim 18 , wherein the game engine is configured to transmit the one or more 3-D transforms to one or more display devices.