Viewport-aware quality metric for 360-degree video
Techniques and systems are provided for processing 360-degree video data. For example, 360-degree video data can be obtained that includes a representation including spherical video data mapped to faces of a geometry. The representation includes a viewport corresponding to an orientation in a 360-degree scene. A window can be determined on a spherical representation of the 360-degree scene at the orientation corresponding to the viewport of the representation. The window is determined based on a front face of the geometry corresponding to the representation. A viewport-aware quality metric can then be determined for the window on the spherical representation of the 360-degree scene.
1. A method of processing 360-degree video data, the method comprising:
obtaining the 360-degree video data, the 360-degree video data including a first representation including spherical video data mapped to faces of a first geometry and a second representation including spherical video data mapped to faces of a second geometry, wherein the first representation is associated with a first viewport corresponding to a first orientation in a 360-degree scene and the second representation is associated with a second viewport corresponding to a second orientation in the 360-degree scene;
determining a first window on a spherical representation of the 360-degree scene at the first orientation corresponding to the first viewport of the first representation, the first window being determined based on a front face of the first geometry;
determining a second window on the spherical representation of the 360-degree scene at the second orientation corresponding to the second viewport of the second representation, the second window being determined based on a front face of the second geometry;
determining a first quality metric of the first window on the spherical representation of the 360-degree scene and a second quality metric of the second window on the spherical representation of the 360-degree scene; and
determining, using at least the first quality metric and the second quality metric, a combined quality metric for the 360-degree video data.
2. The method of claim 1 , wherein determining the combined quality metric includes averaging at least the first quality metric and the second quality metric.
3. The method of claim 1 , wherein the first window and the second window on the spherical representation of the 360-degree scene include front face centers of the first representation and the second representation distributed on the spherical representation of the 360-degree scene.
4. The method of claim 1 , wherein the first window and the second window on the spherical representation of the 360-degree scene are overlapping.
5. The method of claim 1 , wherein determining the first quality metric includes computing a Spherical-Signal-to-Noise ratio (S-PSNR) of video data in the first window.
6. The method of claim 5 , wherein computation of the S-PSNR is restricted to the video data in the first window to determine the first quality metric of the first window.
7. The method of claim 1 , wherein the first geometry includes a first truncated square pyramid (TSP) and the second geometry includes a second TSP, wherein the spherical video data of the first representation is mapped to six faces of the first TSP, and wherein the spherical video data of the second representation is mapped to six faces of the second TSP.
8. The method of claim 1 , wherein the first window is determined by computing a window size for the first window, the window size being computed based on the front face of the first geometry corresponding to the first representation and at least one or more of an amount of separation between the first window and at least one other window of at least one other representation of the 360-degree video data, an estimated head rotational speed of a viewer of the 360-degree video data, or a switching delay associated with switching between representations of the 360-degree video data.
9. The method of claim 1 , wherein computation of the first quality metric is restricted to video data in the first window to determine the first quality metric of the first window.
10. An apparatus for processing 360-degree video data, comprising:
a memory configured to store the 360-degree video data; and
a processor configured to:
obtain the 360-degree video data, the 360-degree video data including a first representation including spherical video data mapped to faces of a first geometry and a second representation including spherical video data mapped to faces of a second geometry, wherein the first representation is associated with a first viewport corresponding to a first orientation in a 360-degree scene and the second representation is associated with a second viewport corresponding to a second orientation in the 360-degree scene;
determine a first window on a spherical representation of the 360-degree scene at the first orientation corresponding to the first viewport of the first representation, the first window being determined based on a front face of the first geometry;
determine a second window on the spherical representation of the 360-degree scene at the second orientation corresponding to the second viewport of the second representation, the second window being determined based on a front face of the second geometry;
determine a first quality metric of the first window on the spherical representation of the 360-degree scene and a second quality metric of the second window on the spherical representation of the 360-degree scene; and
determine, using at least the first quality metric and the second quality metric, a combined quality metric for the 360-degree video data.
11. The apparatus of claim 10 , wherein determining the combined quality metric includes averaging at least the first quality metric and the second quality metric.
12. The apparatus of claim 10 , wherein the first window and the second window on the spherical representation of the 360-degree scene include front face centers of the first representation and the second representation distributed on the spherical representation of the 360-degree scene.
13. The apparatus of claim 10 , wherein the first window and the second window on the spherical representation of the 360-degree scene are overlapping.
14. The apparatus of claim 10 , wherein determining the first quality metric includes computing a Spherical-Signal-to-Noise ratio (S-PSNR) of video data in the first window.
15. The apparatus of claim 14 , wherein computation of the S-PSNR is restricted to the video data in the first window to determine the quality metric of the first window.
16. The apparatus of claim 10 , wherein the first geometry includes a first truncated square pyramid (TSP) and the second geometry includes a second TSP, wherein the spherical video data of the first representation is mapped to six faces of the first TSP, and wherein the spherical video data of the second representation is mapped to six faces of the second TSP.
17. The apparatus of claim 10 , wherein the first window is determined by computing a window size for the first window, the window size being computed based on the front face of the first geometry corresponding to the first representation and at least one or more of an amount of separation between the first window and at least one other window of at least one other representation of the 360-degree video data, an estimated head rotational speed of a viewer of the 360-degree video data, or a switching delay associated with switching between representations of the 360-degree video data.
18. The apparatus of claim 10 , further comprising:
a display for displaying the video data.
19. The apparatus of claim 10 , wherein the apparatus comprises a mobile device with a camera for capturing pictures.
20. The apparatus of claim 10 , wherein computation of the first quality metric is restricted to video data in the first window to determine the first quality metric of the first window.
21. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to:
obtain 360-degree video data, the 360-degree video data including a first representation including spherical video data mapped to faces of a first geometry and a second representation including spherical video data mapped to faces of a second geometry, wherein the first representation is associated with a first viewport corresponding to a first orientation in a 360-degree scene and the second representation is associated with a second viewport corresponding to a second orientation in the 360-degree scene;
determine a first window on a spherical representation of the 360-degree scene at the first orientation corresponding to the first viewport of the first representation, the first window being determined based on a front face of the first geometry;
determine a second window on the spherical representation of the 360-degree scene at the second orientation corresponding to the second viewport of the second representation, the second window being determined based on a front face of the second geometry;
determine a first quality metric of the first window on the spherical representation of the 360-degree scene and a second quality metric of the second window on the spherical representation of the 360-degree scene; and
determine, using at least the first quality metric and the second quality metric, a combined quality metric for the 360-degree video data.
22. The non-transitory computer-readable medium of claim 21 , wherein the first window and the second window on the spherical representation of the 360-degree scene include front face centers of the first representation and the second representation distributed on the spherical representation of the 360-degree scene.
23. The non-transitory computer-readable medium of claim 21 , wherein determining the first quality metric includes computing a Spherical-Signal-to-Noise ratio (S-PSNR) of video data in the first window.
24. The non-transitory computer-readable medium of claim 23 , wherein computation of the S-PSNR is restricted to the video data in the first window to determine the quality metric of the first window.
25. The non-transitory computer-readable medium of claim 21 , wherein the first geometry includes a first truncated square pyramid (TSP) and the second geometry includes a second TSP, wherein the spherical video data of the first representation is mapped to six faces of the first TSP, and wherein the spherical video data of the second representation is mapped to six faces of the second TSP.
26. The non-transitory computer-readable medium of claim 21 , wherein the first window is determined by computing a window size for the first window, the window size being computed based on the front face of the first geometry corresponding to the first representation and at least one or more of an amount of separation between the first window and at least one other window of at least one other representation of the 360-degree video data, an estimated head rotational speed of a viewer of the 360-degree video data, or a switching delay associated with switching between representations of the 360-degree video data.
27. An apparatus for processing 360-degree video data, comprising:
means for obtaining the 360-degree video data, the 360-degree video data including a first representation including spherical video data mapped to faces of a first geometry and a second representation including spherical video data mapped to faces of a second geometry, wherein the first representation is associated with a first viewport corresponding to a first orientation in a 360-degree scene and the second representation is associated with a second viewport corresponding to a second orientation in the 360-degree scene;
means for determining a first window on a spherical representation of the 360-degree scene at the first orientation corresponding to the first viewport of the first representation, the first window being determined based on a front face of the first geometry;
means for determining a second window on the spherical representation of the 360-degree scene at the second orientation corresponding to the second viewport of the second representation, the second window being determined based on a front face of the second geometry;
means for determining a first quality metric of the first window on the spherical representation of the 360-degree scene and a second quality metric of the second window on the spherical representation of the 360-degree scene; and
means for determining, using at least the first quality metric and the second quality metric, a combined quality metric for the 360-degree video data.