Reprojecting holographic video to enhance streaming bandwidth/quality
Improved video compression and video streaming systems and methods are disclosed for environments where camera motion is common, such as cameras incorporated into head-mounted displays. This is accomplished by combining a 3D representation of the shape of the user's environment (walls, floor, ceiling, furniture, etc.), image data, and data representative of changes in the location and orientation (pose) of the camera between successive image frames, thereby reducing data bandwidth needed to send streaming video in the presence of camera motion.
1. A computer system configured for generating image data with compressed video data, comprising:
one or more processors; and
one or more computer-readable media having stored thereon executable instructions that are executable by the one or more processors to cause the computer system to perform operations comprising:
generating, using a previous camera pose and a 3D representation of a shape of a user's environment, a previous frame image as viewed from the previous camera pose;
receiving compressed video data and camera pose data that defines a current camera pose different than the previous camera pose;
generating, using the current camera pose and the 3D representation of the shape of the user's environment, a reprojection of the previous frame image as if viewed from the current camera pose instead of the previous camera pose, the reprojection of the previous frame image being usable to predict a current frame image as viewed from the current camera pose; and
applying the compressed video data to the reprojection of the previous frame image to generate the current frame image, the compressed video data defining differences relative to the reprojection of the previous frame image.
2. The computer system of claim 1 , wherein the operations further comprise:
causing the previous frame image to be rendered on a display device associated with the computer system; and
causing the current frame image to be rendered on the display device.
3. The computer system of claim 1 , wherein the computer system receives the compressed video data from a compressor that generates the compressed video data by calculating and compressing, for a version of the current frame image captured by a camera at the current camera pose, differences between the reprojection of the previous frame image and the version of the current frame image captured by the camera at the current camera pose.
4. The computer system of claim 3 , wherein the compressed video data is received with surface (depth) information associated the 3D representation of the shape of the user's environment, the 3D representation of the shape of the user's environment changing based at least in part on the surface (depth) information.
5. The computer system as recited in claim 1 , wherein the compressed video data is received as a file generated by a compressor.
6. The computer system as recited in claim 5 , wherein the compressor is part of a second computer system that is remotely located from the computer system, the second computer system including a head mounted augmented reality display as well as a camera, the camera comprising a depth camera.
7. The computer system of claim 1 , wherein the previous camera pose has a previous location and/or a previous orientation, and the current camera pose has a current location different than the previous location and/or has a current orientation different than the previous orientation.
8. The computer system of claim 1 , wherein the generating the reprojection of the previous frame image includes:
updating the 3D representation of the shape of the user's environment based on the current camera pose; and
rendering the updated 3D representation to generate the reprojection of the previous frame image.
9. A method for generating image data with compressed video data, the method being implemented by a computer system that includes at least one hardware processor, the method comprising:
generating, using a previous camera pose and a 3D representation of a shape of a user's environment, a previous frame image as viewed from the previous camera pose;
receiving compressed video data and camera pose data that defines a current camera pose different than the previous camera pose;
generating, using the current camera pose and the 3D representation of the shape of the user's environment, a reprojection of the previous frame image as if viewed from the current camera pose instead of the previous camera pose, the reprojection of the previous frame image being usable to predict a current frame image as viewed from the current camera pose; and
applying the compressed video data to the reprojection of the previous frame image to generate the current frame image, the compressed video data defining differences relative to the reprojection of the previous frame image.
10. The method of claim 9 , wherein the method further includes:
causing the previous frame image to be rendered on a display device associated with the computer system; and
causing the current frame image to rendered on the display device.
11. The method of claim 9 , wherein the computer system receives the compressed video data from a compressor that generates the compressed video data by calculating and compressing, for a version of the current frame image captured by a camera at the current camera pose, differences between the reprojection of the previous frame image and the version of the current frame image captured by the camera at the current camera pose.
12. The method of claim 9 , wherein the compressed video data is received with surface (depth) information associated the 3D representation of the shape of the user's environment, the 3D representation of the shape of the user's environment changing based at least in part on the surface (depth) information.
13. The method as recited in claim 9 , wherein the compressed video data is received as a file.
14. The method as recited in claim 13 , wherein the file is received from a separate computer system, the separate computer system including a head mounted augmented reality display as well as a camera, the camera comprising a depth camera.
15. The method of claim 9 , wherein the previous camera pose has a previous location and/or a previous orientation, and the current camera pose has a current location different than the previous location and/or has a current orientation different than the previous orientation.
16. The method of claim 9 , wherein the generating the reprojection of the previous frame image includes:
updating the 3D representation of the shape of the user's environment based on the current camera pose; and
rendering the updated 3D representation to generate the reprojection of the previous frame image.
17. A computer program product comprising one or more computer-readable hardware storage devices having stored thereon executable instructions that are executable by one or more processors of a computer system to configure the computer system to perform operations comprising:
generating, using a previous camera pose and a 3D representation of a shape of a user's environment, a previous frame image as viewed from the previous camera pose;
receiving compressed video data and camera pose data that defines a current camera pose different than the previous camera pose;
generating, using the current camera pose and the 3D representation of the shape of the user's environment, a reprojection of the previous frame image as if viewed from the current camera pose instead of the previous camera pose, the reprojection of the previous frame image being usable to predict a current frame image as viewed from the current camera pose; and
applying the compressed video data to the reprojection of the previous frame image to generate the current frame image, the compressed video data defining differences relative to the reprojection of the previous frame image.
18. The computer program product of claim 17 , wherein the operations further comprise:
causing the previous frame image to be rendered on a display device associated with the computer system; and
causing the current frame image to be rendered on the display device.
19. The computer program product of claim 17 , wherein the computer system receives the compressed video data from a separate computer system that generates the compressed video data by calculating and compressing, for a version of the current frame image captured by a camera at the current camera pose, differences between the reprojection of the previous frame image and the version of the current frame image captured by the camera at the current camera pose.
20. The computer program product of claim 17 , wherein the generating the reprojection of the previous frame image includes:
updating the 3D representation of the shape of the user's environment based on the current camera pose; and
rendering the updated 3D representation to generate the reprojection of the previous frame image.