Multiple listener cloud render with enhanced instant replay
The techniques disclosed herein provide a high fidelity, rich, and engaging experience for spectators of streaming video services. The techniques disclosed herein enable a system to receive, process and, store session data defining activity of a virtual reality environment. The system can generate recorded video data of the session activity along with rendered spatial audio data, e.g., render the spatial audio in the cloud, for streaming of the video data and rendered spatial audio data to one or more computers. The video data and rendered spatial audio data can provide high fidelity video clips of salient activity of a virtual reality environment. In one illustrative example, the system can automatically create a video from one or more camera positions and audio data that corresponds to the camera positions.
1. A computing device, comprising:
a processor;
a memory having computer-executable instructions stored thereupon which, when executed by the processor, cause the computing device to
receive session data defining a virtual reality environment comprising a participant object, the session data allowing a participant to provide a participant input for controlling a location of the participant object and a direction of the participant object,
analyze the session data to determine a level of activity associated with the participant object,
determine a level of activity associated with one or more virtual objects,
select a location and a direction of a virtual camera perspective, wherein the virtual camera perspective is a first-person perspective projecting from the location of the participant object, wherein the direction is towards the one or more virtual objects when the level of activity of one or more virtual objects is greater than the activity level of the participant object, and
generate an output file comprising video data having images from the location and direction of the virtual camera perspective, wherein the output file further comprises audio data for causing an output device to emanate an audio output from a speaker object location that is selected based on the location of the virtual camera perspective, wherein the audio output emanating from the speaker object location models the location and direction of the virtual camera perspective.
2. The computing device of claim 1 , wherein the location of the virtual camera perspective is within a threshold distance from a path of the participant object.
3. The computing device of claim 1 , wherein the virtual camera perspective is a first-person perspective projecting from the location of the participant object when virtual objects within the first-person perspective have a threshold level of activity.
4. The computing device of claim 1 , wherein the virtual camera perspective is a third-person perspective directed toward the participant object and a virtual object when the participant object has a threshold level of activity with the virtual object.
5. The computing device of claim 1 , wherein the virtual camera perspective is a third-person perspective directed toward the participant object when the participant object has a threshold level of activity.
6. The computing device of claim 1 , wherein the virtual camera perspective toggles between a first-person perspective and a third-person perspective based on one or more time periods.
7. The computing device of claim 1 , wherein generating the audio output comprises configuring an audio signal to the output device in accordance with an Ambisonics-based technology, wherein the output signal defines at least one sound field modeling the location of an audio object associated with a stream, wherein data defining the sound field can be interpreted by one or more computing devices for causing the output device to emanate the audio output based on the stream from the speaker object location.
8. A computing device, comprising:
a processor;
a memory having computer-executable instructions stored thereupon which, when executed by the processor, cause the computing device to
receive session data defining a virtual reality environment comprising a participant object, the session data allowing a participant to provide a participant input for controlling a location of the participant object and a direction of the participant object,
analyze the session data to determine a level of activity associated with the participant object and a type of activity associated with the participant object;
determine a level of activity associated with one or more virtual objects,
select a location and a direction of a virtual camera perspective based on the type of activity associated with the participant object, wherein the virtual camera perspective is a first-person perspective projecting from the location of the participant object, wherein the direction is towards the one or more virtual objects when the level of activity of one or more virtual objects is greater than the activity level of the participant object; and
generate an output file comprising video data having images from the location and direction of the virtual camera perspective, wherein the output file further comprises audio data for causing an output device to emanate an audio output from a speaker object location that is selected based on the location of the virtual camera perspective, wherein the audio output emanating from the speaker object location models the location and direction of the virtual camera perspective.
9. The computing device of claim 8 , wherein the location of the virtual camera perspective is within a threshold distance from a path of the participant object.
10. The computing device of claim 8 , wherein the virtual camera perspective is the first-person perspective projecting from the location of the participant object when a first type of activity is detected, wherein the virtual camera perspective is a third-person perspective directed toward the participant object when a second type of activity is detected.
11. The computing device of claim 10 , wherein the second type of activity comprises a threshold level of activity of a virtual reality object and the participant object, wherein the virtual camera perspective causes the generation of the video data depicting the virtual reality object and the participant object.
12. The computing device of claim 10 , wherein the first type of activity comprises movement of the participant object in a predetermined direction.
13. The computing device of claim 12 , wherein the predetermined direction is at least one of a forward direction or a backward direction.
14. The computing device of claim 8 , wherein the virtual camera perspective toggles between a first-person perspective and a third-person perspective based on one or more time periods.
15. The computing device of claim 8 , wherein generating the audio output comprises configuring an audio signal to the output device in accordance with an Ambisonics-based technology, wherein the output signal defines at least one sound field modeling the location of an audio object associated with a stream, wherein data defining the sound field can be interpreted by one or more computing devices for causing the output device to emanate the audio output based on the stream from the speaker object location.
16. A method comprising:
receiving session data defining a virtual reality environment comprising a participant object, the session data allowing a participant to provide a participant input for controlling a location of the participant object and a direction of the participant object,
analyzing the session data to determine a level of activity and an activity type associated with the participant object;
determining a level of activity associated with one or more virtual objects,
selecting a location and a direction of a virtual camera perspective based on the level of activity and the activity type associated with the participant object, wherein the virtual camera perspective is a first-person perspective projecting from the location of the participant object when the level of activity of one or more virtual objects within the first-person perspective is greater than the activity level of the participant object; and
generating an output file comprising video data having images from the location and direction of the virtual camera perspective, wherein the output file further comprises audio data for causing an output device to emanate an audio output from a speaker object location that is selected based on the location of the virtual camera perspective, wherein the audio output emanating from the speaker object location models the location and direction of the virtual camera perspective.
17. The method of claim 16 , wherein the location of the virtual camera perspective is within a threshold distance from a path of the participant object.
18. The method of claim 16 , wherein the virtual camera perspective is a first-person perspective projecting from the location of the participant object when the one or more virtual objects within the first-person perspective have a threshold level of activity.
19. The method of claim 16 , wherein the virtual camera perspective is a third-person perspective directed toward the participant object and the one or more virtual objects when the participant object has a threshold level of activity with the one or more virtual objects.