Low-latency 360 VR streaming system and method for estimating region of interest of user
A low-latency 360 virtual reality (VR) streaming method for estimating a region of interest (ROI) of a user is proposed. The method may include receiving device pose information from a user device, extracting an ROI of a user, based on the device pose information, and generating ROI tile information corresponding to the ROT. The method may also include generating ROI estimation tile information corresponding to a position at which the ROI is to be changed, and requesting, from a streaming server, a tile corresponding to the ROI tile information and the ROI estimation tile information to receive a high-quality tile stream and a low-quality tile stream of full video that includes a low-quality full video tile stream. The method may further include decoding and rendering the high-quality tile stream and the low-quality full video tile stream and providing the rendered video to the user device.
1. A low-latency 360 virtual reality (VR) streaming method for estimating a region of interest (ROI) of a user, the low-latency 360 VR streaming method comprising:
receiving device pose information from a 360 VR device that comprises a user device;
extracting an ROI of a user, based on the device pose information;
generating ROI tile information corresponding to the ROI;
generating ROI estimation tile information corresponding to a position at which the ROI is to be changed;
requesting, from a streaming server, a tile corresponding to the ROI tile information and the ROI estimation tile information to receive a high-quality tile stream and a low-quality tile stream of full video that comprises a low-quality full video tile stream;
decoding and rendering the high-quality tile stream and the low-quality full video tile stream; and
providing the rendered video to the user device.
2. The low-latency 360 VR streaming method of claim 1 , wherein generating the ROI tile information comprises:
calculating center tile information corresponding to a center of the extracted ROI; and
calculating tile group information corresponding to the ROI, based on the center tile information.
3. The low-latency 360 VR streaming method of claim 2 , wherein calculating the tile group information corresponding to the ROI based on the center tile information comprises:
calculating desired amount information about an adjacent tile by using size information about a tile and a resolution of the ROI; and
calculating a tile, corresponding to the desired amount information about the adjacent tile, as tile group information.
4. The low-latency 360 VR streaming method of claim 3 , wherein generating the ROI estimation tile information comprises:
calculating edge position information about the ROI, based on the tile group information; and
generating tiles adjacent to the ROI tile information as ROI estimation tile information when an amount of change in the ROI is greater than or equal to a certain distance per frame with respect to the edge position information.
5. The low-latency 360 VR streaming method of claim 1 , wherein receiving the high-quality tile stream and the low-quality full video tile stream receives the high-quality tile stream and the low-quality full video tile stream of a common media application format (CMAF) chunk unit.
6. The low-latency 360 VR streaming method of claim 5 , wherein receiving the high-quality tile stream and the low-quality full video tile stream allocates an index to each of a plurality of CMAF chunks corresponding to each tile and receives the CMAF chunks in the order of indexes of CMAF chunks alternately for each tile.
7. The low-latency 360 VR streaming method of claim 5 , wherein receiving the high-quality tile stream and the low-quality full video tile stream comprises:
receiving a low-quality full video tile stream from the streaming server;
receiving a high-quality tile stream that comprises an ROI high-quality tile stream corresponding to the ROI tile information; and
receiving a high-quality tile stream that comprises an ROI estimation high-quality tile stream corresponding to the ROI estimation tile information as the reception of the low-quality full video tile stream and the ROI high-quality tile stream is completed.
8. The low-latency 360 VR streaming method of claim 1 , wherein decoding and rendering the high-quality tile stream and the low-quality full video tile stream comprises:
rendering the decoded high-quality tile stream; and
rendering the low-quality full video tile stream in a non-rendering region as the rendering of the decoded high-quality tile stream is completed.
9. The low-latency 360 VR streaming method of claim 8 , wherein decoding and rendering the high-quality tile stream and the low-quality full video tile stream loads a texture used in the rendering by using a two-circular texture buffer.
10. The low-latency 360 VR streaming method of claim 8 , wherein rendering of the decoded high-quality tile stream checks a renderer buffer corresponding to the high-quality tile stream to perform mapping on a texture region corresponding to a tile, and
wherein rendering the low-quality full video tile stream in the non-rendering region scales up the low-quality full video tile stream by a size of a texture in a renderer buffer corresponding to the low-quality full video tile stream and additionally maps the low-quality full video tile stream to a texture region which is the non-rendering region.
11. A low-latency 360 virtual reality (VR) streaming system for estimating a region of interest (ROI) of a user, the low-latency 360 VR streaming system comprising:
a pose information tracking configured to receive device pose information from a 360 VR device that comprises a user device;
a stream reception processor configured to:
extract an ROI of a user, based on the device pose information,
generate ROI tile information corresponding to the ROI and ROI estimation tile information corresponding to a position at which the ROI is to be changed, and
request, from a streaming server, a tile corresponding to the ROI tile information and the ROI estimation tile information to receive a high-quality tile stream and a low-quality tile stream of full video that comprises a low-quality full video tile stream;
a multi-channel decoder configured to decode the high-quality tile stream and the low-quality full video tile stream; and
a simulcast rendering processor configured to perform rendering based on the decoded high-quality tile stream and the decoded low-quality full video tile stream to provide rendered video to the user device.