Exploiting camera depth information for video encoding
The present disclosure is directed a system and method for exploiting camera and depth information associated with rendered video frames, such as those rendered by a server operating as part of a cloud gaming service, to more efficiently encode the rendered video frames for transmission over a network. The method and system of the present disclosure can be used in a server operating in a cloud gaming service to improve, for example, the amount of latency, downstream bandwidth, and/or computational processing power associated with playing a video game over its service. The method and system of the present disclosure can be further used in other applications where camera and depth information of a rendered or captured video frame is available.
1. A method comprising:
processing a chronological sequence of rendered video frames to generate virtual camera information and a depth associated with the sequence of rendered video frames by mapping a two-dimensional point in a region of a first rendered video frame, as viewed through a virtual camera, to a three-dimensional point based on a position and orientation of the virtual camera during rendering of the first video frame and the depth, wherein the virtual camera information indicates a position of the virtual camera and the depth indicates a distance from the virtual camera;
performing motion-compensation prediction based on the generated virtual camera information and depth to determine a motion vector based on a difference in position of a two-dimensional point in the sequence of rendered video frames; and
determining a size of a search window or a size of a search range within the search window based on the depth.
2. The method of claim 1 , further comprising generating the virtual camera and depth information by:
mapping the three-dimensional point to a two-dimensional point in a region of a second rendered video frame, as viewed through the camera, based on a position and orientation of the camera during rendering of the second rendered video frame; and
determining a motion vector based on a difference in position of the two-dimensional point in the first rendered video frame and the two-dimensional point in the second rendered video frame.
3. The method of claim 2 , further comprising determining a position for the search window in the first video frame for encoding a block of pixels in the region of the second video frame based on the motion vector.
4. The method of claim 3 , further comprising searching for a block of pixels within the search range in the search window based on the block of pixels in the region of the second video frame.
5. The method of claim 2 , further comprising:
selecting the two-dimensional point in the region of the first video frame based on a barycenter of the region of the first video frame.
6. The method of claim 2 , further comprising:
determining the depth value as an average depth of pixels in the region of the first video frame from the virtual camera during rendering of the first video frame or as a depth of the three-dimensional point from the virtual camera during rendering of the first video frame.
7. The method of claim 2 , wherein the first video frame and the second video frame are consecutive video frames in a sequence of video frames.
8. The method of claim 1 , further comprising generating the virtual camera and depth information by:
determining quantization parameters to quantize residuals of encoded blocks of pixels in a first video frame based on the depth information.
9. The method of claim 8 , wherein the quantization parameters are determined to have smaller values for encoded blocks of pixels in the first video frame that, on average, depict objects closer to a camera used to render the first video frame than encoded blocks of pixels in the first video frame that, on average, depict objects farther away from the camera.
10. A computer-readable storage device having instructions recorded thereon, execution of which, by a processing device, causes the processing device to:
process, by a 3D rendering API of the processing device, a chronological sequence of rendered video frames to generate virtual camera information and a depth associated with the sequence of rendered video frames by mapping a two-dimensional point in a region of a first rendered video frame, as viewed through a virtual camera, to a three-dimensional point based on a position and orientation of the virtual camera during rendering of the first video frame and the depth, wherein the virtual camera information indicates a position of the virtual camera and the depth indicates a distance from the virtual camera;
perform, by a video encoder of the processing device, motion-compensation prediction based on the generated virtual camera information and depth to determine a motion vector based on a difference in position of a two-dimensional point in the sequence of rendered video frames; and
determine, by the video encoder, a size of a search window or a size of a search range within the search window based on the depth.
11. The computer-readable storage device of claim 10 , further comprising instructions recorded thereon, execution of which, by a processing device, causes the processing to generate the virtual camera information and depth by:
mapping, by the camera and depth information analyzer, the three-dimensional point to a two-dimensional point in a region of a second rendered video frame, as viewed through the camera, based on a position and orientation of the camera during rendering of the second rendered video frame; and
determining, by the video encoder, a motion vector based on a difference in position of the two-dimensional point in the first rendered video frame and the two-dimensional point in the second rendered video frame.
12. The computer-readable storage device of claim 10 , further comprising instructions recorded thereon, execution of which, by a processing device, causes the video encoder to:
quantize residuals of encoded blocks of pixels in a first video frame based on the depth.
13. A method comprising:
mapping a two-dimensional point in a region of a first rendered video frame, as viewed through a virtual camera, to a three-dimensional point based on a position and orientation of the virtual camera during rendering of the first video frame and a depth value, wherein the depth indicates a distance from the virtual camera;
mapping the three-dimensional point to a two-dimensional point in a region of a second rendered video frame, as viewed through the virtual camera, based on a position and orientation of the virtual camera during rendering of a second video frame;
determining a motion vector based on a difference in position of the two-dimensional point in the first rendered video frame and the two-dimensional point in the second rendered video frame;
providing the motion vector to a video encoder to guide a motion compensation prediction process performed by the video encoder; and
determining a size of a search window or a size of a search range within the search window based on a depth associated with the first rendered video frame and the second rendered video frame.
14. The method of claim 13 , wherein the video encoder uses the motion vector to determine a position for the search window in the first rendered video frame to use to encode a block of pixels in the region of the second rendered video frame.
15. The method of claim 14 , wherein the video encoder searches for a block of pixels within the search range in the search window based on the block pixels in the region of the second video frame.
16. The method of claim 13 , wherein the first video frame and the second video frame are consecutive video frames in a sequence of video frames.
17. The method of claim 13 , further comprising generating a virtual camera and depth information based on the camera and depth information associated with the first and second video frames by
determining quantization parameters to quantize residuals of encoded blocks of pixels in the first video frame based on the depth information.
18. An apparatus comprising:
circuitry configured to process a chronological sequence of rendered video frames to generate virtual camera information and a depth associated with the sequence of rendered video frames by mapping a two-dimensional point in a region of a first rendered video frame, as viewed through a virtual camera, to a three-dimensional point based on a position and orientation of the virtual camera during rendering of the first video frame and the depth, wherein the virtual camera information indicates a position of the virtual camera and the depth indicates a distance from the virtual camera;
circuitry configured to perform motion-compensation prediction based on the generated virtual camera information and depth to determine a motion vector based on a difference in position of a two-dimensional point in the sequence of rendered video frames; and
circuitry configured to determine a size of a search window or a size of a search range within the search window.
19. A system comprising:
an end-user device; and
a server comprising:
circuitry configured to process a chronological sequence of rendered video frames to generate virtual camera information and a depth associated with the sequence of rendered video frames by mapping a two-dimensional point in a region of a first rendered video frame, as viewed through a virtual camera, to a three-dimensional point based on a position and orientation of the virtual camera during rendering of the first video frame and the depth, wherein the virtual camera information indicates a position of the virtual camera and the depth indicates a distance from the virtual camera; and
circuitry configured to perform motion-compensation prediction based on the generated virtual camera information and depth to determine a motion vector based on a difference in position of a two-dimensional point in the sequence of rendered video; and
circuitry configured to determine a size of a search window or a size of a search range within the search window based on the depth.
20. The system of claim 19 , wherein the virtual camera and depth are generated at least by mapping a two-dimensional point in a region of a first rendered video frame, as viewed through a camera, to a three-dimensional point based on a position and orientation of the camera during rendering of the first rendered video frame and a depth value.
21. The system of claim 20 , wherein the virtual camera information and depth are generated at least by mapping the three-dimensional point to a two-dimensional point in a region of a second rendered video frame, as viewed through the camera, based on a position and orientation of the camera during rendering of the second rendered video frame.
22. The system of claim 21 , wherein the virtual camera information and depth are generated at least by determining a motion vector based on a difference in position of the two-dimensional point in the first video frame and the two-dimensional point in the second rendered video frame.