Methods and system for generating and displaying 3D videos in a virtual, augmented, or mixed reality environment
Disclosed is an approach for displaying 3D videos in a VR and/or AR system. The 3D videos may include 3D animated objects that escape from the display screen. The 3D videos may interact with objects within the VR and/or AR environment. The 3D video may be interactive with a user such that based on user input corresponding to decisions elected by the user at certain portions of the 3D video such that a different storyline and possibly a different conclusion may result for the 3D video. The 3D video may be a 3D icon displayed within a portal of a final 3D render world.
1. A method of displaying 3D video that extends beyond a surface of a display screen, the method comprising:
identifying a portal in an environment;
identifying a 3D (three-dimensional) video to be displayed via and through the portal, wherein
the 3D video comprises video data, audio data corresponding to the video data, an animation stream, control data corresponding to the animation stream, and
the animation stream corresponds to the audio data and the video data in the 3D video;
generating one or more 3D models corresponding to the 3D video, wherein
the animation stream corresponds to an animation instruction for a 3D model of the one or more 3D models, and
the control data includes instructions that determine when to play which portion of the animation stream;
rendering the 3D video including the one or more 3D models beyond the portal for multiple viewing positions at an appropriate trigger time along with a rendering of the 3D video so that the 3D video is perceived as a 3D representation from the multiple perspectives, wherein
the multiple perspectives are different from a first perspective at which at least a portion of the 3D video was captured.
2. The method of claim 1 , further comprising:
capturing first video data for at least the portion of the 3D video for the first perspective with a first image capturing device for a first eye of a user; and
capturing second video data for the at least the portion of the 3D video for the first perspective with a second image capturing device for a second eye of the user, wherein
the user perceives the first video data and the second video data together as at least one 3D model of the one or more 3D models, and
the 3D video comprises a stereoscopic 3D video, and the portal corresponds to a display screen or a display surface of the display screen.
3. The method of claim 2 , further comprising:
capturing third video data for the at least the portion of the 3D video for a second perspective with a third image capturing device for the first eye; and
capturing fourth video data for the at least the portion of the 3D video for the second perspective with a fourth image capturing device for the second eye, wherein
the user perceives the third video data and the fourth video data together as the at least one 3D model of the one or more 3D models, and
the one or more 3D models are generated with animations, and at least one of the first, the second, the third, or the fourth image capturing device comprises a virtual image capturing device.
4. The method of claim 3 , further comprising displaying the animations of the at least the portion of the 3D video having the one or more 3D models outside of the display screen for the 3D video so that the at least the portion of the 3D video is perceived by the user as a 3D representation that emerges out of or goes into the display screen.
5. The method of claim 4 , wherein the animations of the one or more 3D models are perceived by the user as to exit a planar surface in the 3D video and to come out into the environment in which the user is located.
6. The method of claim 4 , wherein the animations of the one or more 3D models are perceived by a user as to exit a non-planar surface in the 3D video and come out into the environment in which the user is located.
7. The method of claim 1 , further comprising:
rendering the one or more 3D models onto one or more video panes at a same time for the multiple perspectives, wherein at least one video pane of the one or more video panes functions as a background or a backdrop to display at least one of the one or more 3D models; and
displaying the one or more video panes at the same time for the multiple perspectives from which the one or more 3D models are perceived as 3D representations.
8. The method of claim 2 , further comprising rendering and displaying at least a portion of the 3D video relative to a digital representation of a real-world environment in which the user is located, wherein the stereoscopic 3D video is displayed with the one or more 3D models.
9. The method of claim 1 , rendering the 3D video comprising:
determining first depth information from the 3D video;
determining second depth information that corresponds to the first depth information; and
rendering the 3D video based at least in part upon both the first depth information and the second depth information, wherein
the first depth information is included in the 3D video,
the second depth information pertains to a display location of the portal to a user location of a user to which the 3D video is to be presented, and
a depth information is factored into the rendering of the 3D video and the one or more 3D models, the depth information comprising a distance from a user of a virtual and/or augmented reality device to the 3D video and respective one or more 3D models displayed within a virtual and/or augmented reality environment.
10. The method of claim 1 , further comprising:
determining, with at least the control data, a lead-time for loading at least some of the 3D video before the at least some of the 3D video is needed by the animation stream; and
preloading, using the control data, the at least some of the 3D video based at least in part upon the lead-time, wherein the control data instructs a video player to render the one or more 3D models at the appropriate trigger time along with rendering the 3D video.
11. The method of claim 1 , further comprising representing a 3D structure in the 3D video based at least in part upon homogeneity of how the 3D structure is filled, wherein the one or more 3D models are rendered based at least in part on a voxel-based or a volumetric video stream, and a separate 3D structure is rendered based at least in part upon a point-based or polygon-based video stream.
12. A display system for displaying 3D video that extends beyond a surface of a display screen, the system comprising:
an augmented reality head-mounted display system; and
one or more modules for processing data, wherein the one or more modules are stored in one or more memory, the one or more modules configured to perform a set of acts, the set of acts comprising:
identifying a portal in an environment;
identifying a 3D video (three-dimensional) video to be displayed via and through the portal, wherein;
the 3D video comprises video data, audio data corresponding to the video data, an animation stream, control data corresponding to the animation stream, and
the animation stream corresponds to the audio data and the video data in the 3D video;
generating one or more 3D models corresponding to the 3D video, wherein
the animation stream corresponds to an animation instruction for a 3D model of the one or more 3D models, and
the control data includes instructions that determine when to play which portion of the animation stream;
rendering the 3D video including the one or more 3D models beyond the portal for multiple viewing positions at an appropriate trigger time along with a rendering of the 3D video so that the 3D video is perceived as a 3D representation from the multiple perspectives, wherein
the multiple perspectives are different from a first perspective at which at least a portion of the 3D video was captured.
13. The display system of claim 12 , the set of acts further comprising:
capturing first video data for at least the portion of the 3D video for the first perspective with a first image capturing device for a first eye of a user; and
capturing second video data for the at least the portion of the 3D video for the first perspective with a second image capturing device for a second eye of the user, wherein
the user perceives the first video data and the second video data together as at least one 3D model of the one or more 3D models, and
the 3D video comprises a stereoscopic 3D video, and the portal corresponds to a display screen or a display surface of the display screen.
14. The display system of claim 13 , the set of acts further comprising:
capturing third video data for the at least the portion of the 3D video for a second perspective with a third image capturing device for the first eye; and
capturing fourth video data for the at least the portion of the 3D video for the second perspective with a fourth image capturing device for the second eye, wherein
the user perceives the third video data and the fourth video data together as the at least one 3D model of the one or more 3D models, and
the one or more 3D models are generated with animations, and at least one of the first, the second, the third, or the fourth image capturing device comprises a virtual image capturing device.
15. The display system of claim 14 , wherein the set of acts further comprises displaying animations of the at least the portion of the 3D video having the one or more 3D models outside of the display surface for the 3D video so that the at least the portion of the 3D video is perceived by the user as a 3D representation that emerges out of or goes into the display screen.
16. The display system of claim 15 , wherein the animations of the one or more 3D models are perceived by the user as to exit the planar surface in the 3D video when viewed from a second perspective and to come out into the environment in which the user is located.
17. The display system of claim 15 , wherein the animations of the one or more 3D models are perceived by the user as to exit a non-planar surface in the 3D video and to come out into the environment in which the user is located.
18. The display system of claim 12 , wherein the set of acts further comprises:
rendering the one or more 3D models onto one or more video panes at a same time for the multiple perspectives, wherein at least one video pane of the one or more video panes functions as a background or a backdrop to display at least one of the one or more 3D models; and
displaying the one or more video panes at the same time for the multiple perspectives from which the one or more 3D models are perceived as 3D representations.
19. The display system of claim 13 , further comprising rendering and displaying at least a portion of the 3D video relative to a digital representation of a real-world environment in which the user is located, wherein the stereoscopic 3D video is displayed with the one or more 3D models.
20. The display system of claim 12 , rendering the 3D video comprising:
determining first depth information from the 3D video;
determining second depth information that corresponds to the first depth information; and
rendering the 3D video based at least in part upon both the first depth information and the second depth information, wherein
the first depth information is included in the 3D video,
the second depth information pertains to a display location of the portal to a user location of a user to which the 3D video is to be presented, and
a depth information is factored into the rendering of the 3D video and the one or more 3D models, the depth information comprising a distance from a user of a virtual and/or augmented reality device to the 3D video and respective one or more 3D models displayed within a virtual and/or augmented reality environment.
21. The display system of claim 12 , the set of acts further comprising:
determining, with at least the control data, a lead-time for loading at least some of the 3D video before the at least some of the 3D video is needed by the animation stream; and
preloading, using the control data, the at least some of the 3D video based at least in part upon the lead-time, wherein the control data instructs a video player to render the one or more 3D models at the appropriate trigger time along with rendering the 3D video.
22. The display system of claim 12 , further comprising representing a 3D structure in the 3D video based at least in part upon homogeneity of how the 3D structure is filled, wherein the one or more 3D models are rendered based at least in part on a voxel-based or volumetric video stream, and a separate 3D structure is rendered based at least in part upon a point-based or polygon-based video stream.