Priority-based video encoding and transmission
A video encoding system in which pixel data is decomposed into frequency bands prior to encoding. The frequency bands are organized into blocks that are provided to a block-based encoder that encodes the blocks and passes the encoded blocks to a wireless interface that packetizes the blocks for transmittal over a wireless connection. The encoder may categorize the encoded frequency bands into multiple priority levels, and may tag each frequency block with metadata indicating the frequency band represented in the block, the priority of the frequency band, and timing information. The wireless interface may then transmit or drop packets according to the priority levels of the encoded frequency blocks in the packets and/or according to the timing information of the frequency blocks in the packets.
1. A device, comprising:
a display subsystem; and
one or more processors configured to:
receive packets each including one or more blocks, each block including compressed frequency data and a respective instance of timing information;
drop one or more of the blocks from one or more of the packets based on the respective instances of timing information;
decompress the compressed frequency data from the blocks that were not dropped;
reconstruct pixel data for a video frame from the decompressed frequency data; and
provide the reconstructed pixel data to the display subsystem for display.
2. The device as recited in claim 1 , wherein each block further includes a priority level corresponding to a frequency band of the respective compressed frequency data, and wherein, to decompress the compressed frequency data from the blocks that were not dropped, the one or more processors are further configured to decompress the compressed frequency data from the blocks according to the respective priority levels of the blocks.
3. The device as recited in claim 2 , wherein the one or more processors are further configured to prioritize blocks with higher priority levels in said decompress the compressed frequency data, wherein lower frequency bands are assigned higher priority levels, and higher frequency bands are assigned lower priority levels.
4. The device as recited in claim 1 , wherein the timing information includes a deadline for receiving the respective compressed frequency data at the device, and wherein the one or more processors are configured to drop the one or more blocks from the one or more packets upon determining that the deadline for receiving the compressed frequency data in the one or more blocks has expired.
5. The device as recited in claim 1 , wherein the packets are received from a video encoding system and wherein the packets have been generated by the video encoding system by:
decomposing pixel blocks from the video frame into a plurality of frequency bands;
organizing the frequency bands from the pixel blocks into a plurality of blocks each including frequency data for one of the frequency bands;
applying an encoding technique to the blocks to compress the frequency data for each frequency band;
assigning respective instances of timing information to respective ones of the plurality of blocks; and
transmitting packets each including one or more of the blocks to the device.
6. The device as recited in claim 5 , wherein the packets have been further generated by the video encoding system by:
assigning respective priority levels to the frequency bands; and
prioritizing transmission of the blocks of the device according to the assigned priority levels.
7. The device as recited in claim 5 , wherein the video encoding system is a component of a base station.
8. The device as recited in claim 1 , wherein the packets are received via a wireless connection.
9. The device as recited in claim 1 , wherein the device is a head-mounted display (HMD) device.
10. A method, comprising:
one or more processors configured to implement:
receiving packets each including one or more blocks, each block including compressed frequency data and a respective instance of timing information;
dropping one or more of the blocks from one or more of the packets based on the respective instances of timing information;
decompressing the compressed frequency data from the blocks that were not dropped;
reconstructing pixel data for a video frame the decompressed frequency data; and
providing the reconstructed pixel data to a display subsystem for display.
11. The method as recited in claim 10 , further comprising:
assigning priority levels to respective frequency bands of a plurality of frequency bands, wherein lower frequency bands are assigned higher priority levels and higher frequency bands are assigned lower priority levels; and
wherein decompressing the compressed frequency data from the blocks that were not dropped comprises decompressing the compressed frequency data from the blocks according to the assigned priority levels.
12. The method as recited in claim 10 , wherein the respective instances of timing information include corresponding deadlines for receiving the respective compressed frequency data, and wherein dropping one or more of the blocks from one or more of the packets based on the respective timing information comprises dropping the one or more blocks upon determining that the corresponding deadlines for receiving the compressed frequency data in the one or more blocks have expired.
13. The method as recited in claim 10 , wherein the packets are received via a wireless connection.
14. The method as recited in claim 10 , wherein the one or more processors and the display subsystem are components of a head-mounted display (HMD) device.
15. A system, comprising:
a video decoding subsystem comprising one or more processors;
a display subsystem; and
a video encoding subsystem comprising one or more additional processors configured to:
decompose pixel blocks from a video frame into a plurality of frequency bands;
organize the plurality of frequency bands into a plurality of blocks each including frequency data for one of the frequency bands;
apply an encoding technique to the blocks to compress the frequency data for each frequency band;
assign respective instances of timing information to respective ones of the plurality of blocks; and
transmit packets each including one or more of the blocks to the video decoding subsystem;
wherein the one or more processors of the video decoding subsystem are configured to
receive the packets each including one or more blocks, each block including compressed frequency data and a respective instance of timing information;
drop one or more of the blocks from one or more of the packets based on the respective timing information;
decompress the compressed frequency data from the blocks that were not dropped;
reconstruct pixel data for the video frame from the decompressed frequency data; and
provide the reconstructed pixel data to the display subsystem for display.
16. The system as recited in claim 15 , wherein each block further includes a priority level corresponding to a frequency band of the respective compressed frequency data, and wherein, to decompress the compressed frequency data from the blocks that were not dropped, the one or more processors of the video decoding subsystem are further configured to decompress the compressed frequency data from the blocks according to priority levels assigned to the plurality of frequency bands.
17. The system as recited in claim 15 , wherein the video encoding subsystem is further configured to prioritize transmission of the blocks the video decoding subsystem according to priority levels assigned to the plurality of frequency bands.
18. The system as recited in claim 15 , wherein the respective instances of timing information include corresponding deadlines for receiving the respective compressed frequency data at the video decoding subsystem, and wherein the one or more processors of the video decoding subsystem are configured to drop the one or more blocks from the one or more packets upon determining that the corresponding deadlines for receiving the compressed frequency data in the one or more blocks have expired.
19. The system as recited in claim 15 , wherein the packets are transmitted via a wireless connection.
20. The system as recited in claim 15 , wherein the system is a virtual or mixed reality (VR/MR) system.