Providing segmentation information for immersive video
Systems and methods are provided in which a plurality of source views of a scene are encoded as an immersive video comprising a plurality of patches. The patches are segmented based on segmentation information from at least one of the source views. The segmentation information associates each of a plurality of regions of the source view with a respective entity. Information is also encoded that indicates which of the source views are associated with the segmentation information used to segment the patches. This information may be used to provide improved rendering, particularly where rendering is performed of less than all entities in the scene.
1 . A method comprising:
obtaining a plurality of source views of a scene;
for at least one of the source views, obtaining segmentation information associating each of a plurality of regions of the source view with a respective entity;
encoding the plurality of source views as an immersive video comprising a plurality of patches, the patches being segmented according to the segmentation information; and
encoding information indicating which of the source views are associated with the segmentation information used to segment the patches.
2 . The method of claim 1 , wherein the information indicating which of the source views are associated with the segmentation information used to segment the patches comprises, for each source view, a flag indicating whether that input source view is associated with segmentation information used to segment the patches.
3 . The method of claim 1 , wherein the segmentation information associated with a source view comprises an entity map associated with the source view.
4 . The method of claim 1 , wherein the information indicating which of the source views are associated with the segmentation information used to segment the patches comprises information indicating whether the segmentation information is based on a depth image or on a texture image.
5 . A method comprising:
obtaining an encoded immersive video comprising a plurality of patches, the video representing a plurality of input views of a scene;
obtaining information indicating which of the input views are associated with segmentation information used to segment the patches; and
rendering the immersive video according to the information indicating which of the input views are associated with segmentation information used to segment the patches.
6 . The method of claim 5 , wherein the information indicating which of the input views are associated with the segmentation information used to segment the patches comprises, for each input view, a flag indicating whether that input view is associated with segmentation information used to segment the patches.
7 . The method of claim 5 , wherein the information indicating which of the input views are associated with the segmentation information used to segment the patches comprises information indicating whether the segmentation information is based on a depth image or on a texture image.
8 . The method of claim 5 , wherein rendering the immersive video according to the information indicating which of the input views are associated with segmentation information used to segment the patches comprises:
identifying at least one selected entity to be rendered; and
in response to a determination that the segmentation information is based on a depth image, performing warping of depth pixels of the immersive video only for depth patches that are associated with the at least one selected entity.
9 . The method of claim 5 , wherein rendering the immersive video according to the information indicating which of the input views are associated with segmentation information used to segment the patches comprises:
identifying at least one selected entity to be rendered; and
in response to a determination that the segmentation information is based on a texture image:
performing warping of depth pixels of the immersive video for depth patches including at least depth patches that are associated with the at least one selected entity; and
performing blending of color values based only on color pixels that are associated with the at least one selected entity.
10 . The method of claim 5 , wherein rendering the immersive video according to the information indicating which of the input views are associated with segmentation information used to segment the patches comprises:
in response to a determination that the segmentation information is based on a depth image, making a visibility determination based at least in part on the segmentation information.
11 . An apparatus comprising one or more processors configured to perform:
obtaining a plurality of source views of a scene;
for at least one of the source views, obtaining segmentation information associating each of a plurality of regions of the source view with a respective entity;
encoding the plurality of source views as an immersive video comprising a plurality of patches, the patches being segmented according to the segmentation information; and
encoding information indicating which of the source views are associated with the segmentation information used to segment the patches.
12 . The apparatus of claim 11 , wherein the information indicating which of the source views are associated with the segmentation information used to segment the patches comprises, for each source view, a flag indicating whether that source view is associated with segmentation information used to segment the patches.
13 . The apparatus of claim 11 , wherein the segmentation information associated with a source view comprises an entity map associated with the source view.
14 . The apparatus of claim 11 , wherein the information indicating which of the source views are associated with the segmentation information used to segment the patches comprises information indicating whether the segmentation information is based on a depth image or on a texture image.
15 . An apparatus comprising one or more processors configured to perform:
obtaining an encoded immersive video comprising a plurality of patches, the video representing a plurality of input views of a scene;
obtaining information indicating which of the input views are associated with segmentation information used to segment the patches; and
rendering the immersive video according to the information indicating which of the input views are associated with segmentation information used to segment the patches.
16 . The apparatus of claim 15 , wherein the information indicating which of the input views are associated with the segmentation information used to segment the patches comprises, for each input view, a flag indicating whether that input view is associated with segmentation information used to segment the patches.
17 . The apparatus of claim 15 , wherein the information indicating which of the input views are associated with the segmentation information used to segment the patches comprises information indicating whether the segmentation information is based on a depth image or on a texture image.
18 . The apparatus of claim 15 , wherein rendering the immersive video according to the information indicating which of the input views are associated with segmentation information used to segment the patches comprises:
identifying at least one selected entity to be rendered; and
in response to a determination that the segmentation information is based on a depth image, performing warping of depth pixels of the immersive video only for depth patches that are associated with the at least one selected entity.
19 . The apparatus of claim 15 , wherein rendering the immersive video according to the information indicating which of the input views are associated with segmentation information used to segment the patches comprises:
identifying at least one selected entity to be rendered; and
in response to a determination that the segmentation information is based on a texture image:
performing warping of depth pixels of the immersive video for depth patches including at least depth patches that are associated with the at least one selected entity; and
performing blending of color values based only on color pixels that are associated with the at least one selected entity.
20 . The apparatus of claim 15 , wherein rendering the immersive video according to the information indicating which of the input views are associated with segmentation information used to segment the patches comprises:
in response to a determination that the segmentation information is based on a depth image, making a visibility determination based at least in part on the segmentation information.