Efficient capture and delivery of walkable and interactive virtual reality or 360 degree video
Disclosed are systems and methods for generating a walkable 360-degree video or virtual reality (VR) environment. 360-degree video data is obtained for a real-world environment and comprises a plurality of chronologically ordered frames captured by traversing a first path through the real-world environment. One or more processing operations are applied to generate a processed 360-degree video, which can be displayed to a user of an omnidirectional treadmill. Locomotion information is received from one or more sensors of the omnidirectional treadmill, wherein the locomotion information is generated based on a physical movement on or within the omnidirectional treadmill. Using the received locomotion information, one or more playback commands for controlling playback of the processed 360-degree video are generated. One or more selected frames of the processed 360-degree video are rendered for presentation and display to the user, based on the one or more playback commands.
1. A method for generating a virtual reality environment, the method comprising:
obtaining a first 360-degree video data of a real-world environment, wherein the first 360-degree video data comprises a first plurality of chronologically ordered frames captured by traversing at least a first path through the real-world environment;
processing the first 360-degree video data the to generate a first processed 360-degree video of a plurality of processed 360-degrees videos;
obtaining a second 360-degree video data of the real-world environment, wherein the second 360-degree video data comprises a second plurality of chronologically ordered frames captured by traversing a second path through the real-world environment;
processing the second 360-degree video data to generate a second processed 360-degree video of the plurality of processed 360-degree videos, wherein the second processed 360-degree video data and the first processed 360-degree video intersect at a junction point corresponding to a location where the second path intersects the first path;
receiving locomotion information from one or more sensors of an omnidirectional treadmill, wherein the locomotion information is generated based on a physical movement within the omnidirectional treadmill;
generating, using the received locomotion information, one or more playback commands for controlling playback of the plurality of processed 360-degree videos; and
rendering one or more frames of the plurality of processed 360-degree videos based on the one or more playback commands.
2. The method of claim 1 , further comprising generating, based on the plurality of processed 360-degree videos, a walkable virtual reality (VR) environment, the walkable VR environment including a first or second virtual path based on the first or second path through the real-world environment.
3. The method of claim 2 , further comprising:
calculating a direction of virtual movement along the first virtual path within the walkable VR environment, the calculating based on the received locomotion information;
obtaining the one or more frames from the plurality of processed 360-degree videos in response to a determination that the calculated direction of virtual movement corresponds to a traversal direction in which the first 360-degree video data was captured along the first path through the real-world environment.
4. The method of claim 2 , further comprising:
calculating a direction of virtual movement along the second virtual path within the walkable VR environment, the calculating based on the received locomotion information;
obtaining the one or more frames from the plurality of processed 360-degree videos in response to a determination that the calculated direction of virtual movement corresponds to a traversal direction in which the second 360-degree video data was captured along the second path through the real-world environment.
5. The method of claim 1 , wherein the received locomotion information comprises one or more of:
a direction of locomotion or a speed of locomotion on the omnidirectional treadmill.
6. The method of claim 5 , wherein the one or more playback commands comprise adjustments to a playback speed of the plurality of processed 360-degree videos, where an elapsed time between rendering the one or more frames is proportional to the speed of locomotion.
7. The method of claim 1 , wherein processing the first or second 360-degree video data to generate the first or second processed 360-degree video data comprises performing one or more of:
stabilization;
top and tailing;
cropping;
adjusting playback speed; and
mirroring.
8. A virtual reality (VR) system comprising:
a processor;
a virtual reality rendering engine;
a virtual reality display; and
a memory having instructions therein, which when executed by the processor cause the processor to:
obtain a first 360-degree video data of a real-world environment, wherein the first 360-degree video data comprises a plurality of chronologically ordered frames captured by traversing at least a first path through the real-world environment;
process the first 360-degree video data to generate a first processed 360-degree video of a plurality of processed 360-degrees videos;
obtain a second 360-degree video data of the real-world environment, wherein the second 360-degree video data comprises a second plurality of chronologically ordered frames captured by traversing a second path through the real-world environment;
process the 360-degree video data to generate a second processed 360-degree video of the plurality of processed 360-degrees videos, wherein the second processed 360-degree video data and the first processed 360-degree video intersect at a junction point corresponding to a location where the second path intersects the first path;
receive locomotion information from one or more sensors of an omnidirectional treadmill, wherein the locomotion information is generated based on a physical movement within the omnidirectional treadmill;
generate, using the received locomotion information, one or more playback commands for controlling playback of the plurality processed 360-degree videos; and
render, using the virtual reality rendering engine, one or more frames of the plurality of processed 360-degree videos based on the one or more playback commands.
9. The system of claim 8 , wherein the received locomotion information comprises at least one of a direction of locomotion or a speed of locomotion, on the omnidirectional treadmill.
10. The system of claim 8 , wherein:
the one or more playback commands comprise adjustments to a playback speed of the plurality of processed 360-degree videos, where an elapsed time between rendering the one or more frames is proportional to the speed of locomotion.
11. The system of claim 8 , wherein the instructions further cause the processor to generate the first or second processed 360-degree video data by performing one or more of:
stabilization;
top and tailing;
cropping;
adjusting playback speed; and
mirroring.
12. The system of claim 8 , wherein the instructions further cause the processor to generate, based on the plurality of processed 360-degree videos, a walkable virtual reality (VR) environment, the walkable VR environment including a first or second virtual path based on the first or second path through the real-world environment.
13. The system of claim 12 , wherein the instructions further cause the processor to:
calculate a direction of virtual movement along the second virtual path within the walkable VR environment, the calculating based on the received locomotion information;
obtain the one or more frames from the plurality of processed 360-degree videos in response to a determination that the calculated direction of virtual movement corresponds to a traversal direction in which the second 360-degree video data was captured along the second path through the real-world environment.
14. The system of claim 12 , wherein the instructions further cause the processor to:
calculate a direction of a virtual movement along the first virtual path within the first 360-degree video, the calculating based on the received locomotion information;
obtain the one or more frames from the plurality of processed 360-degree videos in response to a determination that the calculated direction of virtual movement corresponds to a traversal direction in which the first 360-degree video data was captured along the first path through the real-world environment.