Communication protocol for streaming mixed-reality environments between multiple devices
An immersive content presentation system can capture the motion or position of a performer in a real-world environment. A game engine can be modified to receive the position or motion of the performer and identify predetermined gestures or positions that can be used to trigger actions in a 3-D virtual environment, such as generating a digital effect, transitioning virtual assets through an animation graph, adding new objects, and so forth. The use of the 3-D environment can be rendered and composited views can be generated. Information for constructing the composited views can be streamed to numerous display devices in many different physical locations using a customized communication protocol. Multiple real-world performers can interact with virtual objects through the game engine in a shared mixed-reality experience.
1. A method comprising:
receiving, at a server, changes to a virtual asset in a 3-D virtual environment, wherein the 3-D virtual environment is shared between a plurality of real-world environments;
preparing, by the server, a stream of real-time communication packets, wherein each communication packet in the stream of communication packets comprises:
a first field configured to store a 2-D image frame depicting at least in part a view of a first real-world environment in the plurality of real-world environments and/or a rendered view of the 3-D virtual environment; and
a second field configured to represent one or more 3-D transforms to be applied to the virtual asset in the 3-D virtual environment by devices receiving the stream of real-time communication packets; and
transmitting, from the server, the stream of real-time communication packets to a plurality of display devices distributed in the plurality of real-world environments.
2. The method of claim 1 , wherein each communication packet in the stream of communication packets further comprises a timestamp.
3. The method of claim 1 , wherein each communication packet in the stream of communication packets further comprises a location indicator.
4. The method of claim 1 , wherein each communication packet in the stream of communication packets further comprises a spatial anchor.
5. The method of claim 1 , wherein the plurality of display devices comprises one or more pairs of AR glasses.
6. The method of claim 5 , wherein the first field comprises a rendered view of the 3-D virtual environment to be displayed on the one or more pairs of AR glasses.
7. The method of claim 5 , wherein the first field is empty.
8. An immersive content presentation system comprising:
one or more processors; and
one or more memory devices comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving, at a server, changes to a virtual asset in a 3-D virtual environment, wherein the 3-D virtual environment is shared between a plurality of real-world environments;
preparing, by the server, a stream of real-time communication packets, wherein each communication packet in the stream of communication packets comprises:
a first field configured to store a 2-D image frame depicting at least in part a view of a first real-world environment in the plurality of real-world environments and/or a rendered view of the 3-D virtual environment; and
a second field configured to represent one or more 3-D transforms to be applied to the virtual asset in the 3-D virtual environment by devices receiving the stream of real-time communication packets; and
transmitting, from the server, the stream of real-time communication packets to a plurality of display devices distributed in the plurality of real-world environments.
9. The immersive content presentation system of claim 8 , wherein the plurality of display devices comprises one or more pairs of VR goggles.
10. The method of claim 9 , wherein the first field comprises a rendered view of the first real-world environment composited with a rendered view of the 3-D virtual environment to be displayed on the one or more pairs of VR goggles.
11. The method of claim 9 , wherein the second field is empty.
12. The immersive content presentation system of claim 8 , wherein the plurality of display devices comprises one or more mobile phones with cameras.
13. The method of claim 12 , wherein the first field comprises a rendered view of the 3-D virtual environment to be composited on the one or more pairs of VR goggles.
14. The method of claim 8 , wherein the contents of the first field and the contents of the second field are populated based on the corresponding display devices in the plurality of display devices to which the stream of real-time communication packets is transmitted.
15. A non-transitory, computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving, at a server, changes to a virtual asset in a 3-D virtual environment, wherein the 3-D virtual environment is shared between a plurality of real-world environments;
preparing, by the server, a stream of real-time communication packets, wherein each communication packet in the stream of communication packets comprises:
a first field configured to store a 2-D image frame depicting at least in part a view of a first real-world environment in the plurality of real-world environments and/or a rendered view of the 3-D virtual environment; and
a second field configured to represent one or more 3-D transforms to be applied to the virtual asset in the 3-D virtual environment by devices receiving the stream of real-time communication packets; and
transmitting, from the server, the stream of real-time communication packets to a plurality of display devices distributed in the plurality of real-world environments.
16. The non-transitory, computer-readable medium of claim 15 , wherein the 3-D virtual environment is rendered at the server.
17. The non-transitory, computer-readable medium of claim 15 , wherein the 3-D virtual environment is rendered at the plurality of display devices.
18. The non-transitory, computer-readable medium of claim 15 , wherein the view of the first real-world environment and the view of the 3-D virtual environment are composited at the server.
19. The non-transitory, computer-readable medium of claim 15 , wherein the view of the first real-world environment and the view of the 3-D virtual environment are composited at the plurality of display devices.
20. The non-transitory, computer-readable medium of claim 15 , wherein the one or more 3-D transforms are received from a game engine.