Cinematic space-time view synthesis for enhanced viewing experiences in computing environments
A mechanism is described for facilitating cinematic space-time view synthesis in computing environments according to one embodiment. A method of embodiments, as described herein, includes capturing, by one or more cameras, multiple images at multiple positions or multiple points in times, where the multiple images represent multiple views of an object or a scene, where the one or more cameras are coupled to one or more processors of a computing device. The method further includes synthesizing, by a neural network, the multiple images into a single image including a middle image of the multiple images and representing an intermediary view of the multiple views.
1. An apparatus comprising:
memory;
instructions; and
processor circuitry to execute the instructions to:
access images of a scene for which actual calibration information and actual position information is not used, different ones of the images captured by different cameras, different ones of the images captured at different times, different ones of the images captured from different first perspectives relative to the scene, different ones of the images including objects moving in the scene;
synthesize, with a neural network, interpolated images based on different ones of the images, the interpolated images representative of perspectives of the scene corresponding to second perspectives relative to the scene, the second perspectives different from the first perspectives, the synthesis of the interpolated images not dependent on synchronization of the different times of the different images; and
generate a video of the scene based on the interpolated images.
2. The apparatus of claim 1 , wherein the images are RGB images that do not include depth information.
3. The apparatus of claim 1 , wherein the neural network is one of at least two neural networks.
4. The apparatus of claim 3 , wherein the at least two neural networks are part of a single network.
5. The apparatus of claim 1 , wherein the actual calibration information and the actual position information for the images is not used to synthesize the interpolated images.
6. The apparatus of claim 1 , wherein the video of the scene provides a smoother view of the scene than is possible using the different ones of the images without the interpolated images.
7. The apparatus of claim 1 , wherein the video corresponds to a view of the scene that follows a virtual camera track that moves relative to the scene.
8. At least one storage device comprising instructions to cause at least one processor circuit to at least:
access images of a scene for which actual calibration information and actual position information is not used, different ones of the images captured by different cameras, different ones of the images captured at different times, different ones of the images captured from different first perspectives relative to the scene, different ones of the images including objects moving in the scene;
synthesize, with a neural network, interpolated images based on different ones of the images, the interpolated images representative of perspectives of the scene corresponding to second perspectives relative to the scene, the second perspectives different from the first perspectives, the synthesis of the interpolated images not dependent on synchronization of the different times of the different images; and
generate a video of the scene based on the interpolated images.
9. The at least one storage device of claim 8 , wherein the images are RGB images that do not include depth information.
10. The at least one storage device of claim 8 , wherein the neural network is one of at least two neural networks.
11. The at least one storage device of claim 10 , wherein the at least two neural networks are part of a single network.
12. The at least one storage device of claim 8 , wherein the actual calibration information and the actual position information for the images is not used to synthesize the interpolated images.
13. The at least one storage device of claim 8 , wherein the video of the scene provides a smoother view of the scene than is possible using the different ones of the images without the interpolated images.
14. The at least one storage device of claim 8 , wherein the video corresponds to a view of the scene that follows a virtual camera track that moves relative to the scene.
15. An apparatus comprising:
means for storing information; and
means for executing instructions, the executing means to:
access images of a scene for which at least one of actual calibration information or actual position information is not used, different ones of the images captured by different cameras, different ones of the images captured at different times, different ones of the images captured from different first angles relative to the scene, different ones of the images including objects moving in the scene;
synthesize, with a neural network, interpolated images based on different ones of the images, the interpolated images representative of perspectives of the scene corresponding to second angles relative to the scene, the second angles different from the first angles, the synthesis of the interpolated images not based on the different times of the different images being synchronized; and
generate a video of the scene based on the interpolated images.
16. The apparatus of claim 15 , wherein the images are RGB images that do not include depth information.
17. The apparatus of claim 15 , wherein the neural network is one of at least two neural networks.
18. The apparatus of claim 17 , wherein the at least two neural networks are part of a single network.
19. The apparatus of claim 15 , wherein the actual calibration information and the actual position information for the images is not used to synthesize the interpolated images.
20. The apparatus of claim 15 , wherein the video of the scene provides a smoother view of the scene than can be provided using the different ones of the images without the interpolated images.
21. The apparatus of claim 15 , wherein the video corresponds to a view of the scene that follows a virtual camera track that moves relative to the scene.