Ad-hoc dynamic capture of an immersive virtual reality experience
Methods, systems, and techniques for projecting streamed video are provided. An Example Surround Video Projection System (SVPS) provides support for interacting with the displayed virtual reality environment, capturing a video of a user in the VR environment, and displaying the VR environment on deformations. Also, the SVPS may change the displayed VR environment responsive to attributes and/or characteristics of the user. In one example, the SVPS comprises a real time, interactive rendering system, a display system, and one or more display units. The rendering system comprises a high resolution graphics engine capable of generating high resolution video. The projection system comprises video capture cards to capture the generated video stream and forward it to a projection mapping engine. The projection mapping engine consolidates and stitches together the received video stream as appropriate to render the video stream over display units to the target viewing environment such as a dome/sphere.
1. A system, comprising:
a surround video projection environment comprising a plurality of display surfaces that define an interior space inside of which one or more elements are located, wherein the display surfaces comprise a plurality of perforations, wherein at least one video camera of a plurality of video cameras is situated behind each of the perforations, wherein the video cameras are configured to record first video of the interior space while the surround video projection environment displays a second video of a virtual environment on the display surfaces; and
a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, performs a method comprising:
receiving the first video recorded by the video cameras;
determining volumetric capture data of the first video;
processing, using the second video, the volumetric capture data to isolate the one or more elements in the volumetric capture data;
after processing the volumetric capture data, generating a composited video stream that includes a composite of the second video and the volumetric capture data; and
storing the composited video stream.
2. The system of claim 1 , wherein the non-transitory computer-readable storage medium is a component of the surround video projection environment.
3. The system of claim 1 wherein the second video comprises an equirectangular map, wherein the composited video stream comprises the equirectangular map composited with the volumetric capture data.
4. The system of claim 1 wherein processing the volumetric capture data to isolate the one or more elements, comprises: subtracting out the displayed virtual environment from the volumetric capture data.
5. The system of claim 1 , further comprising: using a depth sensor to acquire depth measurements for each of the elements, wherein the depth measurements are used to subtract out the displayed virtual environment.
6. The system of claim 1 wherein processing the volumetric capture data to isolate the one or more elements, comprises: removing, from the volumetric capture data, projections of the displayed virtual environment onto each of the one or more elements in the volumetric capture data.
7. The system of claim 1 , wherein the method further comprises: transmitting the composited video stream to a virtual reality headset.
8. The system of claim 7 , wherein the virtual reality headset is external to the surround video projection environment.
9. The system of claim 7 , wherein transmitting the composited video stream comprises streaming the composited video stream to the virtual reality headset in real-time as the second video of the virtual environment is displayed on the display surfaces.
10. A computer-implemented method, comprising:
displaying first video of a virtual environment onto a plurality of display surfaces of a surround video projection environment, the plurality of display surfaces defining an interior space inside of which one or more elements are located;
capturing, while the first video is displayed, and using a plurality of cameras situated behind perforations in the display surfaces, second video of the interior space;
determining volumetric capture data of the second video;
processing, using the first video, the volumetric capture data to isolate the one or more elements in the volumetric capture data;
after processing the volumetric capture data, generating a composited video stream that includes a composite of the first video and the volumetric capture data; and
forwarding the composited video stream.
11. The method of claim 10 , wherein forwarding the composited video stream comprises forwarding the composited video stream to a virtual reality headset for playback.
12. The method of claim 10 , wherein the first video of the virtual environment is a 360 degree video.
13. The method of claim 10 , wherein processing the volumetric capture data to isolate the one or more elements in the volumetric capture data comprises: subtracting out the displayed virtual environment from the volumetric capture data.
14. The method of claim 13 , wherein processing the volumetric capture data to isolate the one or more elements in the volumetric capture data comprises: removing, from the volumetric capture data, projections of the displayed virtual environment onto each of the one or more elements in the volumetric capture data.
15. The method of claim 10 , wherein processing the volumetric capture data to isolate the one or more elements in the volumetric capture data comprises: removing, from the volumetric capture data, projections of the displayed virtual environment onto each of the one or more elements in the volumetric capture data.
16. A non-transitory computer-readable medium including instructions that, when executed by one or more processors, perform a method comprising:
receiving first video recorded by a plurality of video cameras, each of the video cameras located behind at least one of a plurality of display surfaces in a video projection environment, the plurality of display surfaces defining an interior space inside of which one or more elements are located, the first video comprising video of the interior space, wherein the first video is recorded while the surround video projection environment displays a second video of a virtual environment on the display surfaces;
determining volumetric capture data of the first video;
processing, using the second video, the volumetric capture data to isolate the one or more elements in the volumetric capture data;
after processing the volumetric capture data, generating a composited video stream that includes a composite of the second video and the volumetric capture data; and
storing the composited video stream.
17. The non-transitory computer-readable medium of claim 16 , wherein processing the volumetric capture data to isolate the one or more elements, comprises: subtracting out the displayed virtual environment from the volumetric capture data.
18. The non-transitory computer-readable medium of claim 17 , wherein processing the volumetric capture data to isolate the one or more elements, comprises: removing, from the volumetric capture data, projections of the displayed virtual environment onto each of the one or more elements in the volumetric capture data.
19. The non-transitory computer-readable medium of claim 16 , wherein the second video comprises an equirectangular map, wherein the composited video stream comprises the equirectangular map composited with the volumetric capture data.