Reprojecting holographic video to enhance streaming bandwidth/quality
Improved video compression and video streaming systems and methods are disclosed for environments where camera motion is common, such as cameras incorporated into head-mounted displays. This is accomplished by combining a 3D representation of the shape of the user's environment (walls, floor, ceiling, furniture, etc.), image data, and data representative of changes in the location and orientation (pose) of the camera between successive image frames, thereby reducing data bandwidth needed to send streaming video in the presence of camera motion.
1. A computer system for compressing video data, comprising:
one or more processors; and
one or more computer-readable media having stored thereon executable instructions that when executed by the one or more processors configure the computer system to perform at least the following:
capture successive frames of video from a camera incorporated into a first device, the successive frames of video comprising a previous frame image and a current frame image and including surface (depth) information;
construct a first 3D representation of a shape of a user's environment based on the surface (depth) information of the previous frame image and a first camera location and orientation (pose) of the camera corresponding to the previous frame image;
project the previous frame image onto the first 3D representation;
detect a change in the location and orientation (pose) of the camera between the previous frame image and the current frame image and generating current frame camera pose data representative of the location and orientation (pose) of the camera for the current frame image;
construct a second 3D representation of the shape of the user's environment based on a second camera location and orientation (pose) corresponding to the current frame image, the second 3D representation corresponding to the shape of the user's environment as viewed from the second location and orientation (pose) of the camera, and render the second 3D representation to generate a re-projected previous frame image as viewed from the second location and orientation (pose) of the camera;
pass the re-projected previous frame image to a video compressor for computing differences between the re-projected previous frame image and the current frame image; and
generate compressed video data comprising only the differences between the re-projected previous frame image and the current frame image.
2. The computer system as recited in claim 1 , wherein the camera comprises a depth camera and the first device comprises a head mounted augmented reality display.
3. The computer system as recited in claim 2 , wherein the processing unit is further configured to save the compressed video data to a file.
4. The computer system of claim 2 , wherein the processing unit is further configured to communicate to a second device the compressed video data and the current frame camera pose data.
5. The computer system of claim 4 , wherein the processing unit is further configured to:
receive at the second device, the compressed video data and the current frame camera pose data;
construct, by the second device, a 3D representation of the shape of the user's environment based on the received current frame camera pose data and render the 3D representation to generate the re-projected previous frame image data; and
apply the received compressed video data to the re-projected previous frame image data to generate current frame image data.
6. The computer system of claim 5 , wherein the processing unit is further configured to save the current frame image data to a file.
7. The computer system of claim 5 , wherein the processing unit is further configured to render the current image data on a display associated with the second device.
8. The computer system of claim 5 , wherein the processing unit is further configured to transmit changes in the surface (depth) data.
9. The computer system of claim 8 , wherein the processing unit is further configured to accumulate, for all previous image frames, a textured model of the 3D environment to refine the textures applied to surfaces as a user encounters the 3D environment from different camera positions over time.
10. A video compression method comprising:
capturing successive frames of video from a camera incorporated into a first device, the successive frames of video comprising a previous frame image and a current frame image and including surface (depth) information;
constructing a first 3D representation of a shape of a user's environment based on the surface (depth) information of the previous frame image and a first camera location and orientation (pose) of the camera corresponding to the previous frame image;
projecting the previous frame image onto the first 3D representation;
detecting a change in the location and orientation (pose) of the camera between the previous frame image and the current frame image and generating current frame camera pose data representative of the location and orientation (pose) of the camera for the current frame image;
constructing a second 3D representation of the shape of the user's environment based on a second camera location and orientation (pose) corresponding to the current frame image, the second 3D representation corresponding to the shape of the user's environment as viewed from the second location and orientation (pose) of the camera, and rendering the second 3D representation to generate a re-projected previous frame image as viewed from the second location and orientation (pose) of the camera;
passing the re-projected previous frame image to a video compressor for computing differences between the re-projected previous frame image and the current frame image; and
generating compressed video data comprising only the differences between the re-projected previous frame image and the current frame image.
11. The method of claim 10 , the method further comprising:
an act of saving the compressed video data to a file.
12. The method of claim 10 , the method further comprising: an act for communicating to a second device the compressed video data and the current frame camera pose data.
13. The method of claim 12 , the method further comprising acts of: receiving at the second device, the compressed video data and the current frame camera pose data;
rendering, by the second device, a 3D representation of the shape of the user's environment based on the received current frame camera pose data;
generating the re-projected previous frame image data; and
applying the received compressed video data to the re-projected previous frame image data to generate current frame image data.
14. The method of claim 13 , the method further comprising an act for saving the current frame image data to a file.
15. The method of claim 13 , the method further comprising: an act for rendering the current image data on a display associated with the second device.
16. The method of claim 10 , the method further comprising: detecting any new surfaces or features in the 3D representation of the shape of a user's environment that become visible from the location and orientation (pose) of the camera for the current frame image and generating 3D representation change data representative of the differences between the 3D representation of the user's environment associated with the previous image frame and the 3D representation of the user's environment associated with the current image frame.
17. The method of claim 16 , the method further comprising, an act for communicating to a second device the compressed video data, the current frame camera pose data and the 3D representation change data.
18. A video compression method comprising:
capturing successive frames of video from a camera incorporated into a first device, the successive frames of video comprising a previous frame image and a current frame image and including surface (depth) information;
rendering the depth of pixels in the previous frame image for a first location into a z-buffer;
detecting a change in the location and orientation (pose) of the camera between the previous frame image and the current frame image and generating current frame camera pose data representative of a second location and orientation (pose) of the camera for the current frame image;
performing a screenspace transformation operation, wherein each pixel of the image has color (from the camera) and depth (from the z-buffer), to re-project the previous frame image as viewed from the second location and orientation (pose) of the camera based on the 3D position of each pixel;
passing the re-projected previous frame image to a video compressor for computing differences between the re-projected previous frame image and the current frame image; and
generating compressed video data comprising only the differences between the re-projected previous frame image and the current frame image.
19. The method of claim 18 , the method further comprising: an act for rendering the current image data on a display associated with a second device.
20. One or more hardware storage devices comprising stored computer-executable instructions that are executable by one or more processors for implementing a method for streaming video from a forward-facing depth camera mounted to the front of a head mounted augmented reality display, wherein the method includes:
capturing successive frames of video from a camera incorporated into a first device, the successive frames of video comprising a previous frame image and a current frame image and including surface (depth) information;
constructing a first 3D representation of a shape of a user's environment based on the surface (depth) information of the previous frame image and a first camera location and orientation (pose) of the camera corresponding to the previous frame image;
projecting the previous frame image onto the first 3D representation;
detecting a change in the location and orientation (pose) of the camera between the previous frame image and the current frame image and generating current frame camera pose data representative of the location and orientation (pose) of the camera for the current frame image;
constructing a second 3D representation of the shape of the user's environment based on a second camera location and orientation (pose) corresponding to the current frame image, the second 3D representation corresponding to the shape of the user's environment as viewed from the second location and orientation (pose) of the camera, and rending the second 3D representation to generate a re-projected previous frame image as viewed from the second location and orientation (pose) of the camera;
passing the re-projected previous frame image to a video compressor for computing differences between the re-projected previous frame image and the current frame image; and
generating compressed video data comprising only the differences between the re-projected previous frame image and the current frame image.