Low latency wireless virtual reality systems and methods
Virtual Reality (VR) systems, apparatuses and methods of processing data are provided which include predicting, at a server, a user viewpoint of a next frame of video data based on received user feedback information sensed at a client, rendering a portion of the next frame using the prediction, encoding the portion, formatting the encoded portion into packets and transmitting the video data. At a client, the encoded and packetized A/V data is received and depacketized. The portion of video data and corresponding audio data is decoded and controlled to be displayed and aurally provided in synchronization. Latency may be minimized by utilizing handshaking between hardware components and/or software components such as a 3D server engine, one or more client processors, one or more client processors, a video encoder, a server NIC, a video decoder, a client NIC; and a 3D client engine.
1. A method of processing Virtual Reality (VR) data, the method comprising:
receiving user feedback information;
using one or more server processors to:
predict, based on the user feedback information, a user viewpoint of a next frame of a sequence of frames of video data to be displayed;
render a portion of the next frame of video data to be displayed using the prediction; and
encode the portion of the next frame of video data to be displayed; and
transmit the encoded and formatted portion of the next frame of video data to be displayed.
2. The method of claim 1 , further comprising using the one or more server processors to wirelessly transmit the encoded and formatted portion of the next frame of video data of video data to be displayed.
3. The method of claim 2 , further comprising using the one or more server processors to:
provide rendering instructions to render the portion of the next frame of video data to be displayed using the prediction;
encode audio data corresponding to the portion of the next frame of video data using the prediction;
combine the encoded audio data and encoded video data to encoded audio-video (A/V) data;
format the encoded A/V data into one or more packets; and
wirelessly transmit the combined encoded A/V data via a wireless channel dedicated to the encoded A/V data.
4. The method of claim 1 , wherein the user feedback information comprises tracking information indicating at least one of a position and orientation of a portion of a user; and
the tracking information is received via a wireless channel dedicated to the tracking information.
5. The method of claim 1 , wherein the feedback information includes time stamp information indicating a time when the feedback information is sensed prior to be received and further comprising using the one or more server processors to:
determine when the portion of the next frame of video data to be displayed is rendered;
indicate that the portion of the next frame of video data is rendered; and
encode the portion of the frame of video data in response to the indication.
6. The method of claim 1 , further comprising using one or more server processors to:
receive, at a server network interface controller (NIC), the encoded and formatted portion of the next frame of video data via direct memory access (DMA), and
transmit, via the server NIC, the encoded and formatted portion of the next frame of video data via a channel dedicated to the tracking information.
7. A processor device for processing Virtual Reality (VR) content, the processor device comprising:
memory configured to store data; and
one or more processor configured to:
predict, based on received user feedback information, a user viewpoint of a next frame of a sequence of frames of video data to be displayed;
render a portion of the next frame of video data to be displayed using the prediction;
encode the portion of the next frame of video data to be displayed; and
control transmission of the encoded portion of the next frame of video data to be displayed.
8. The processor device of claim 7 , wherein the one or more client processors are further configured to wirelessly transmit the encoded and formatted portion of the next frame of video data.
9. The processor device of claim 8 , further comprising using the one or more server processors to:
provide rendering instructions to render the portion of the next frame of video data to be displayed using the prediction;
encode audio data corresponding to the portion of the next frame of video data using the prediction;
combine the encoded audio and video (A/V) data;
format the encoded A/V data into packetized A/V data; and
wirelessly transmit the combined encoded and packetized A/V data via a wireless channel dedicated to the encoded and packetized A/V data.
10. The processor device of claim 7 , wherein the one or more server processors are further configured to:
determine when the portion of the next frame of video data to be displayed is rendered;
indicate that the portion of the next frame of video data is rendered; and
encode the portion of the frame of video data in response to the indication.
11. The processor device of claim 7 , wherein the user feedback information comprises tracking information indicating at least one of a position and orientation of a portion of a user; and
the tracking information is received via a wireless channel dedicated to the tracking information.
12. The processor device of claim 7 , further comprising:
an audio encoder configured to encode audio data corresponding to the sequence of frames of video data;
a video encoder configured to encode the portion of the next frame of video data; and
a multiplexer configured to combine the decoded A/V data.