Asynchronous time and space warp with determination of region of interest
A method and a system for warping a rendered frame is disclosed. On a host device of a split-rendering system, the method includes generating the rendered frame based on head tracking information of a user. The method also includes identifying a region of interest (ROI) of the rendered frame. The method also includes generating metadata for a warping operation from the ROI. The method further include transmitting the rendered frame and the metadata for a warping operation of the rendered frame. On a client device of the split-rendering system, the method includes transmitting head tracking information of a user by a client device. The method also includes receiving the rendered frame and metadata. The method further includes warping the rendered frame using the metadata and display pose information. The host device and the client device may be combined into an all-in-one head mounted display.
1. A method for warping a rendered frame by a client device comprising:
transmitting eye tracking information of a user, wherein the eye tracking information is used to determine a region of interest (ROI);
transmitting head tracking information of the user;
receiving a rendered frame and metadata, wherein the rendered frame is based on the head tracking information and the metadata is based on the ROI of the rendered frame, wherein the metadata comprises a single depth approximation of a plurality of pixel depths of pixels within the ROI of the rendered frame;
determining display pose information of the user; and
warping the rendered frame based on the metadata and the display pose information.
2. The method of claim 1 , further comprising:
analyzing content information of the rendered frame within the ROI;
generating a motion vector filter kernel size based on analyzing the content information of the rendered frame within the ROI;
receiving a plurality of motion vectors of the rendered frame;
filtering the plurality of motion vectors of the rendered frame using the motion vector filter kernel size to generate warp vectors; and
warping the rendered frame based on the warp vectors.
3. The method of claim 1 , wherein warping the rendered frame based on the metadata and the display pose information comprises:
warping the rendered frame to take into account the single depth approximation of the pixels within the ROI based on a translation of the client device determined from the head tracking information and the display pose information; and
warping the rendered frame independent of the single depth approximation of the pixels within the ROI based on a rotation of the client device determined from the head tracking information and the display pose information.
4. The method of claim 1 , wherein warping the rendered frame based on the metadata and the display pose information comprises:
generating potentially visible triangles in the rendered frame for vector streaming only for the ROI of the rendered frame based on the metadata; and
warping the rendered frame based on the potentially visible triangles in the rendered frame.
5. An apparatus, comprising:
a memory storing processor readable code; and
a processor coupled to the memory and configured to execute the processor readable code to cause the apparatus to:
transmit eye tracking information of a user, wherein the eye tracking information is used to determine a region of interest (ROI);
transmit head tracking information of the user;
receive a rendered frame and metadata, wherein the rendered frame is based on the head tracking information and the metadata is based on the ROI of the rendered frame, wherein the metadata comprises a single depth approximation of a plurality of pixel depths of pixels within the ROI of the rendered frame;
determine display pose information of the user; and
warp the rendered frame based on the metadata and the display pose information.
6. The apparatus of claim 5 , wherein the processor when executing the processor readable code further causes the apparatus to:
analyze content information of the rendered frame within the ROI;
generate a motion vector filter kernel size based on the content information analyzed;
receive a plurality of motion vectors of the rendered frame;
filter the plurality of motion vectors of the rendered frame using the motion vector filter kernel size to generate warp vectors; and
warp the rendered frame based on the warp vectors.
7. The apparatus of claim 5 , wherein to warp the rendered frame based on the metadata and the display pose information, the processor when executing the processor readable code further causes the apparatus to:
warp the rendered frame to take into account the single depth approximation of the pixels within the ROI based on a translation of the apparatus determined from the head tracking information and the display pose information; and
warp the rendered frame independent of the single depth approximation of the pixels within the ROI based on a rotation of the apparatus determined from the head tracking information and the display pose information.
8. The apparatus of claim 5 , wherein to warp the rendered frame based on the metadata and the display pose information, the processor when executing the processor readable code further cause the apparatus to:
generate potentially visible triangles in the rendered frame for vector streaming only for the ROI of the rendered frame based on the metadata; and
warp the rendered frame based on the potentially visible triangles in the rendered frame.