Video encoding system
A video encoding system in which pixel data is decomposed into frequency bands prior to encoding. The frequency bands are organized into blocks that are provided to a block-based encoder. The encoded frequency data is packetized and transmitted to a receiving device. On the receiving device, the encoded data is decoded to recover the frequency bands. Wavelet synthesis is then performed on the frequency bands to reconstruct the pixel data for display. The system may encode parts of frames (tiles or slices) using one or more encoders and transmit the encoded parts as they are ready. A pre-filter component may perform a lens warp on the pixel data prior to the wavelet transform.
1. A video encoding system, comprising:
one or more processors; and
a memory, storing program instructions that when executed by the one or more processors, cause the one or more processors to implement:
a wavelet transform component configured to:
apply a wavelet transform to respective ones of a plurality of pixel blocks to decompose the respective ones of the pixel blocks into a plurality of frequency bands;
organize the plurality of frequency bands into a plurality of frequency blocks, wherein respective ones of the frequency blocks include frequency data corresponding to respective ones of the plurality of frequency bands; and
provide respective ones of the frequency blocks as separate streams to two or more encoders, wherein respective ones of the streams correspond to a respective one of the frequency bands; and
the two or more encoders, wherein respective ones of the two or more encoders are configured to:
process the separate streams of frequency blocks, wherein:
to process the frequency blocks in a given stream, the respective one of the two or more encoders is further configured to apply an encoding technique to the frequency blocks to compress the frequency data in the frequency blocks; and
the processing of the frequency blocks in the given stream is multiplexed; and
provide the compressed frequency data from the separate streams to a wireless interface for transmittal over a wireless connection to a receiving device.
2. The video encoding system of claim 1 , wherein the multiplexed processing allows multiple time units to be encoded in the processing of respective frequency blocks corresponding to a given frequency band.
3. The video encoding system of claim 1 , wherein the multiplexed processing of the frequency blocks is based, at least in part, on a dependency of a second frequency block on a prior processing of a first frequency block.
4. The video encoding system of claim 1 , wherein the program instructions further cause the one or more processors to implement:
a pre-filter component configured to:
pre-filter the pixel blocks, wherein the pixel blocks are from a rendered video frame, wherein to pre-filter the pixel blocks, the pre-filter component is further configured to apply a filter to the rendered video frame to correct for distortion that will be introduced at the receiving device.
5. The video encoding system of claim 4 , wherein the one or more processors are further configured to:
receive, over the wireless connection, lens warp data, for lenses of the receiving device, to be used by the pre-filter component to correct for the distortion that will be introduced by the lenses of the receiving device.
6. The video encoding system of claim 4 , wherein, to pre-filter the pixel blocks, the pre-filter component is further configured to:
apply a filter to a peripheral region of the rendered video frame to reduce resolution in the peripheral region while maintaining higher resolution in a foveated region of the rendered video frame.
7. The video encoding system of claim 1 , wherein the applied wavelet transform is a two-level wavelet decomposition applied to each pixel block from a rendered video frame.
8. The video encoding system of claim 1 , wherein the applied wavelet transform is an N-level wavelet decomposition applied to each pixel block from a rendered video frame, where N is adjustable.
9. The video encoding system of claim 1 , wherein, to apply the wavelet transform, the wavelet transform component is further configured to:
apply a coring function to one or more wavelet coefficients of a wavelet decomposition.
10. A method, comprising:
applying a wavelet transform to respective ones of a plurality of pixel blocks to decompose the respective ones of the pixel blocks into a plurality of frequency bands;
organizing the plurality of frequency bands into a plurality of frequency blocks, wherein respective ones of the frequency blocks include frequency data corresponding to respective ones of the plurality of frequency bands;
providing respective ones of the frequency blocks as separate streams to two or more encoders, wherein respective ones of the streams correspond to a respective one of the frequency bands;
processing, via multiplexing, the separate streams of frequency blocks, wherein:
processing the frequency blocks in a given stream comprises applying at a respective one of the two or more encoders an encoding technique to the frequency blocks to compress the frequency data in the frequency blocks; and
the processing of the frequency blocks in the given stream is multiplexed; and
transmitting the compressed frequency data from the separate streams to a receiving device over a wireless connection.
11. The method of claim 10 , wherein the processing, via the multiplexing, allows multiple time units to be encoded in the processing of respective frequency blocks corresponding to a given frequency band.
12. The method of claim 10 , wherein the processing, via the multiplexing, is based, at least in part, on a dependency of a second frequency block on a first frequency block.
13. The method of claim 10 , further comprising:
pre-filtering the pixel blocks, wherein the pixel blocks are from a rendered video frame, wherein said pre-filtering comprises applying a filter to the rendered video frame to correct for distortion that will be introduced at the receiving device.
14. The method of claim 13 , further comprising:
receiving lens warp data for lenses of the receiving device,
wherein said applying the filter to the rendered video frame to correct for the distortion that will be introduced at the receiving device is based, at least in part, on the received lens warp data.
15. A system, comprising:
a device comprising one or more processors and a display subsystem; and
a transmitting device comprising:
one or more processors configured to:
apply a wavelet transform to respective ones of a plurality of pixel blocks comprising pixel data to decompose the pixel data into a plurality of frequency bands;
organize the plurality of frequency bands into a plurality of frequency blocks, wherein respective ones of the frequency blocks include frequency data corresponding to respective ones of the plurality of frequency bands; and
provide respective ones of the frequency blocks as separate streams to two or more encoders, wherein respective ones of the streams correspond to a respective one of the frequency bands; and
the two or more encoders, wherein respective ones of the two or more encoders are configured to process the separate streams of frequency blocks, wherein:
to process the frequency blocks in a given stream, the respective one of the two or more encoders is further configured to apply an encoding technique to the frequency blocks in the given stream to compress the frequency bands; and
the processing of the frequency blocks in the given stream is multiplexed; and
a wireless interface configured to transmit the compressed frequency bands over a wireless connection to the device; and
wherein the one or more processors of the device are configured to:
decompress the compressed frequency bands received from the transmitting device;
perform wavelet synthesis on the decompressed frequency bands to reconstruct the pixel data for a video frame; and
provide the reconstructed pixel data to the display subsystem for display.
16. The system of claim 15 , wherein the multiplexed processing allows multiple time units to be encoded in the processing of respective frequency blocks corresponding to a given frequency band.
17. The system of claim 15 , wherein the multiplexed processing of the frequency blocks is based, at least in part, on a dependency of a second frequency block on a processing of a first frequency block.
18. The system of claim 15 , wherein the one or more processors are further configured to:
pre-filter the pixel blocks, wherein to pre-filter comprises applying a filter to correct for distortion that will be introduced at the display subsystem.
19. The system of claim 18 , wherein the one or more processors are further configured to:
receive, over the wireless connection, lens warp data, for lenses of the device, to be used in said pre-filtering the pixel blocks to correct for the distortion that will be introduced by the lenses of the device.
20. The system of claim 15 , wherein the device is a head-mounted display (HMD) device.