Dual detail encoding in a distributed system
Described herein are, among other things, techniques for implementing a dual detail encoding scheme in a distributed display system. At the host computer, an application may render a frame for a scene at a first resolution. The host computer may generate a two-dimensional (2D) array of pixels for the frame, encode the 2D array of pixels, and transmit the encoded pixel data to the display device. The 2D array of pixels comprises, for each eye, a copy of the frame downscaled to a second resolution less than the first resolution, and a copy of a subregion of the scene. At the display device, the encoded pixel data is decoded to obtain the 2D array of pixels for the frame, which is modified by enlarging the copies of the frame so that images can be generated using the enlarged copies of the frame and the copies of the subregion with blending.
1 . A host computer comprising:
a processor; and
memory storing computer-executable instructions that, when executed by the processor, cause the host computer to:
execute a graphics-rendering application to render a frame for a scene at a first resolution;
generate a two-dimensional (2D) array of pixels for the frame, the 2D array of pixels comprising:
first pixels representing a first copy of the frame downscaled to a second resolution less than the first resolution;
second pixels representing a second copy of the frame downscaled to the second resolution;
third pixels representing a first copy of a subregion of the scene; and
fourth pixels representing a second copy of the subregion;
encode the third pixels of the 2D array of pixels to obtain first encoded pixel data representing the first copy of the subregion;
transmit the first encoded pixel data to a display device while the first pixels of the 2D array of pixels are still being encoded;
encode the first pixels of the 2D array of pixels to obtain second encoded pixel data representing the first copy of the frame; and
transmit the second encoded pixel data to the display device.
2 . The host computer of claim 1 , wherein:
the third pixels correspond one-to-one with original pixels of the frame that represent the subregion; and
the fourth pixels correspond one-to-one with the original pixels.
3 . The host computer of claim 1 , wherein:
the third pixels represent the first copy of the subregion downscaled in accordance with an integer ratio; and
the fourth pixels represent the second copy of the subregion downscaled in accordance with the integer ratio.
4 . The host computer of claim 1 , wherein the subregion is predetermined.
5 . The host computer of claim 1 , wherein the computer-executable instructions, when executed by the processor, further cause the host computer to determine the subregion based at least in part on eye tracking data received from the display device.
6 . The host computer of claim 1 , wherein generating the 2D array of pixels comprises arranging the pixels for the frame in a vertical packing layout having:
the third pixels at a top of the 2D array of pixels;
the first pixels below the third pixels;
the fourth pixels below the first pixels; and
the second pixels at a bottom of the 2D array of pixels.
7 . The host computer of claim 1 , wherein the computer-executable instructions, when executed by the processor, further cause the host computer to:
allocate a first number of bits to encode the first pixels and the second pixels; and
allocate a second number of bits to encode the third pixels and the fourth pixels,
wherein the first number of bits is less than the second number of bits.
8 . The host computer of claim 1 , further comprising a transceiver, wherein transmitting the first encoded pixel data to the display device comprises wirelessly transmitting, via the transceiver, one or more first data packets carrying the first encoded pixel data, and wherein transmitting the second encoded pixel data to the display device comprises wirelessly transmitting, via the transceiver, one or more second data packets carrying the second encoded pixel data.
9 . A method comprising:
rendering, by a graphics-rendering application executing on a host computer, a frame for a scene at a first resolution;
generating a two-dimensional (2D) array of pixels for the frame, the 2D array of pixels comprising:
first pixels representing a first copy of the frame downscaled to a second resolution less than the first resolution;
second pixels representing a second copy of the frame downscaled to the second resolution;
third pixels representing a first copy of a subregion of the scene; and
fourth pixels representing a second copy of the subregion;
encoding the third pixels of the 2D array of pixels to obtain first encoded pixel data representing the first copy of the subregion;
transmitting the first encoded pixel data to a display device while the first pixels of the 2D array of pixels are still being encoded;
encoding the first pixels of the 2D array of pixels to obtain second encoded pixel data representing the first copy of the frame; and
transmitting the second encoded pixel data to the display device.
10 . The method of claim 9 , wherein:
the third pixels correspond one-to-one with original pixels of the frame that represent the subregion; and
the fourth pixels correspond one-to-one with the original pixels.
11 . The method of claim 9 , wherein the subregion is predetermined and corresponds to a center of the scene.
12 . The method of claim 9 , further comprising determining the subregion based at least in part on eye tracking data received from the display device.
13 . The method of claim 9 , wherein the generating of the 2D array of pixels comprises arranging the pixels for the frame in a vertical packing layout having:
the third pixels at a top of the 2D array of pixels;
the first pixels below the third pixels;
the fourth pixels below the first pixels; and
the second pixels at a bottom of the 2D array of pixels.
14 . The method of claim 9 , further comprising:
allocating a first number of bits to encode the first pixels and the second pixels; and
allocating a second number of bits to encode the third pixels and the fourth pixels,
wherein the first number of bits is less than the second number of bits.
15 . The method of claim 9 , wherein the transmitting of the first encoded pixel data to the display device comprises wirelessly transmitting, via a transceiver of the host computer, one or more first data packets carrying the first encoded pixel data, and wherein the transmitting of the second encoded pixel data to the display device comprises wirelessly transmitting, via the transceiver, one or more second data packets carrying the second encoded pixel data.
16 . The method of claim 9 , wherein:
the subregion is a rectangle with M pixels in a horizontal dimension and N pixels in a vertical dimension; and
the first copy of the frame and the second copy of the frame are each downscaled to M pixels in the horizontal dimension and N pixels in the vertical dimension.
17 . The method of claim 9 , wherein the first pixels, the second pixels, the third pixels, and the fourth pixels are arranged in a vertical packing layout of four sub-arrays that make up the 2D array of pixels, each of the four sub-arrays having M pixels in a horizontal dimension and N pixels in a vertical dimension.
18 . The method of claim 9 , wherein the first copy of the subregion and the second copy of the subregion are not downscaled.
19 . The host computer of claim 1 , wherein:
the subregion is a rectangle with M pixels in a horizontal dimension and N pixels in a vertical dimension; and
the first copy of the frame and the second copy of the frame are each downscaled to M pixels in the horizontal dimension and N pixels in the vertical dimension.
20 . The host computer of claim 1 , wherein the first pixels, the second pixels, the third pixels, and the fourth pixels are arranged in a vertical packing layout of four sub-arrays that make up the 2D array of pixels, each of the four sub-arrays having M pixels in a horizontal dimension and N pixels in a vertical dimension.