Methods and system for generating and displaying 3D videos in a virtual, augmented, or mixed reality environment
Disclosed is an approach for displaying 3D videos in a VR and/or AR system. The 3D videos may include 3D animated objects that escape from the display screen. The 3D videos may interact with objects within the VR and/or AR environment. The 3D video may be interactive with a user such that based on user input corresponding to decisions elected by the user at certain portions of the 3D video such that a different storyline and possibly a different conclusion may result for the 3D video. The 3D video may be a 3D icon displayed within a portal of a final 3D render world.
1. A computer implemented method of displaying an environment aware 3D video in a virtual and/or augmented reality environment, the method comprising:
identifying a 3D video;
receiving, from one or more sensors, environment information of a user environment, the environment information identifying objects within the environment; and
rendering a portion of the 3D video onto one or more objects identified from the environment,
wherein the 3D video has a video file format comprising control data, wherein the control data instructs a video player to display the portion of the 3D video onto the one or more objects identified from the environment, and
wherein the control data comprise a set of control commands for controlling objects and functions within the 3D video and external to the 3D video.
2. The method of claim 1 , wherein the 3D video is a stereoscopic 3D video.
3. The method of claim 1 , wherein the environment is a physical environment of the user.
4. The method of claim 1 , wherein the one or more sensors comprise one or more cameras for capturing scene information of the physical environment.
5. The method of claim 1 , further comprising:
interpreting scene information captured from the one or more sensors; and
mapping one or more elements of the environment by detecting and registering the one or more elements from the environment.
6. The method of claim 1 , wherein the control commands (e.g., the set of control commands) is at least one of: extensible, updateable, or dynamic.
7. The method of claim 6 , wherein the control commands comprise at least one of: changing the speed of the 3D video, providing a user prompt for dimming lighting in user environment at a beginning of the 3D video, dialing a pizza restaurant at a beginning of the 3D video, or playing the 3D video differently based at least in part on a user's eye gaze.
8. The method of claim 1 , wherein the 3D video is a 3D movie, the 3D movie comprising programming language.
9. A virtual and/or augmented reality display system comprising:
a camera to capture a virtual and/or augmented reality environment; and
a module for processing data, wherein the module is stored in a memory, the module when executed, performs a process of:
identifying a 3D video,
receiving, from one or more sensors, environment information of a user environment, the environment information identifying objects within the environment, and
rendering a portion of the 3D video onto one or more objects identified from the environment,
wherein the 3D video has a video file format comprising control data, and
wherein the control data comprise a set of control commands for controlling objects and functions within the 3D video and external to the 3D video.
10. The virtual and/or augmented reality display system of claim 9 , wherein the 3D video is a stereoscopic 3D video.
11. The virtual and/or augmented reality display system of claim 9 , wherein the control data instructs a video player to render portions of the 3D video based at least in part on decisions elected by the user.
12. The virtual and/or augmented reality display system of claim 9 , wherein the one or more interactions received from the user comprise at least one of a head pose, an eye-tracking, gestures, totem gestures, or object recognizer.
13. The virtual and/or augmented reality display system of claim 9 , wherein an interaction from the one or more interactions received from the user jumps the 3D video to a different part of the 3D video.
14. A computer implemented method of displaying interactive 3D video in a virtual and/or augmented reality environment, the method comprising:
identifying a 3D video as a stream of non-executable data periodically interrupted by decision nodes;
receiving one or more interactions from a user as input corresponding to decisions elected by the user; and
rendering the 3D video based at least on the decisions elected by the user,
wherein the 3D video has a video file format comprising control data, and
wherein the control data comprise a set of control commands for controlling objects and functions within the 3D video and external to the 3D video.
15. The method of claim 14 , wherein the 3D video is a stereoscopic 3D video.
16. The method of claim 14 , wherein the control data instructs a video player to render portions of the 3D video based at least in part on the decisions elected by the user.
17. The method of claim 14 , wherein the one or more interactions received from the user comprise at least one of a head pose, an eye-tracking, gestures, totem gestures, or object recognizer.
18. The method of claim 14 , wherein an interaction from the one or more interactions received from the user jumps the 3D video to a different part of the 3D video.