Representing volumetric video in saliency video streams
Saliency regions are identified in a global scene depicted by volumetric video. Saliency video streams that track the saliency regions are generated. Each saliency video stream tracks a respective saliency region. A saliency stream based representation of the volumetric video is generated to include the saliency video streams. The saliency stream based representation of the volumetric video is transmitted to a video streaming client.
1. A method for streaming volumetric video, comprising:
identifying a set of one or more saliency regions in a global scene depicted by the volumetric video;
generating a set of one or more saliency video streams that track the set of one or more saliency regions identified in the global scene, wherein each saliency video stream in the set of one or more saliency video streams tracks a respective saliency region in the set of one or more saliency regions;
generating a saliency stream based representation of the volumetric video, wherein the saliency stream based representation includes the set of one or more saliency video streams;
transmitting the saliency stream based representation of the volumetric video to a video streaming client,
wherein the set of one or more saliency video streams includes a first saliency video stream assigned with a first saliency rank and a second saliency video stream assigned with a second saliency rank lower than the first saliency rank, wherein the second saliency video stream is removed from the set of one or more saliency video streams to be transmitted to the video streaming client at a later time, in response to determining that an available data rate has been reduced.
2. The method of claim 1 , wherein the first saliency rank and/or the second saliency rank is ranked by one or more of: input from a content creator or a director, viewer statistical information gathering and analyses, and visual saliency algorithms based at least in part on computer vision techniques.
3. The method of claim 1 , wherein the saliency stream based representation includes at least one base video stream depicting the global scene including image areas other than the set of one or more saliency regions, wherein the at least one base video stream enables the video streaming client to render imagery outside the set of one or more saliency regions depicted by the set of one or more saliency video streams.
4. The method of claim 1 , wherein the saliency stream based representation includes a disocclusion data associated with a saliency video stream in the set of one or more saliency video streams, wherein the disocclusion data includes texture and depth information for image details occluded in a reference view depicted by the saliency video stream, wherein the image details occluded in the reference view become disoccluded in one or more other views adjacent to the reference view.
5. The method of claim 1 , wherein the set of one or more saliency video streams includes the first saliency video stream assigned with a first saliency rank and the second saliency video stream assigned with a second saliency rank lower than the first saliency rank, wherein the second saliency video stream is removed from the set of one or more saliency video streams to be transmitted to the video streaming client at a later time, in response to determining, based on real time viewpoint data, that a virtual view represented by a viewer's viewpoint is directed to a different spatial region away from a saliency region tracked by the second saliency video stream.
6. The method of claim 1 , wherein image metadata is transmitted with the set of one or more saliency video streams to enable the video streaming client to render images derived from the set of saliency video streams and any accompanying disocclusion.
7. The method of claim 1 , wherein viewpoint data of a viewer collected in real time while the viewer is viewing imagery generated from the volumetric video is received from the video streaming client, wherein the viewpoint data is used to select the set of one or more saliency streams in one or more reference views closest to a virtual view represented by a viewpoint, of the viewer, as indicated in the viewpoint data.
8. The method of claim 1 , wherein an input version of the volumetric video is received and used to derive the saliency stream based representation of the volumetric video, wherein the input version of the volumetric video includes a group of pictures (GOP) comprising a plurality of full resolution images depicting the global scene, wherein at least one saliency video stream in the set of one or more saliency video streams is initialized at a starting time point of the GOP.
9. An apparatus comprising a non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors cause performance of the method recited in claim 1 .
10. A non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors cause performance of the method recited in claim 1 .
11. A computing device comprising one or more processors and one or more storage media, storing a set of instructions, which when executed by one or more processors cause performance of the method recited in claim 1 .