Granted Patent B2

US 12,395,616 · App. 18/736,571 · Granted Aug 19, 2025

Inventors: Serhan Gül (Berlin, DE); Yago Sánchez De La Fuente (Berlin, DE); Cornelius Hellge (Berlin, DE); Thomas Schierl (Berlin, DE); Robert Skupin (Berlin, DE); Thomas Wiegand (Berlin, DE)

Assignee: Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V.

H04N13/194H04N13/161H04N13/172H04N13/332H04N13/366H04N21/21805H04N21/41407H04N21/6587H04N21/816

Abstract

Claims (57)

1. An apparatus for a 360° video communication with a sender, wherein

the apparatus is to

acquire from the sender video data dependent on a certain viewing direction of a 360° video, and

display the video data representing the certain viewing direction of the 360° video,

wherein, for the certain viewing direction of the 360° video, the apparatus is to trigger a switching during the session between

(i) first video data rendered by the sender and representing a 2D video at the certain viewing direction of the 360° video; and

(ii) second video data not rendered by the sender and representing at least a part of the 360° video to be transmitted by the sender using a certain projection.

2. The apparatus of claim 1 , wherein, to acquire the video data from the sender, the apparatus is to

signal to the sender the certain viewing direction of the 360° video, and

receive from the sender the video data for the certain viewing direction of the 360° video.

3. The apparatus of claim 1 , wherein the apparatus comprises

a display device to display to a user the video data for the certain viewing direction of the 360° video,

a sensor to detect the viewing direction of the user, and

a processor to signal the detected viewing direction to the sender and to process the received video data for display on the display device.

4. The apparatus of claim 1 , wherein, during the session of the 360° video communication, the apparatus is to request from the sender the first video data or the second video data dependent on an end-to-end latency between the receiver and the sender.

5. The apparatus of claim 1 , wherein

at the beginning of a session of the 360° video communication, the apparatus is to negotiate with the sender, and

when negotiating with the sender, the apparatus is to receive from the sender Supplementary Enhancement Information (SEI) messages indicating one or more of a projection type, a rotation and region-wise packing (RWP) constraints.

6. The apparatus of claim 1 , wherein the apparatus comprises a viewport predictor providing a viewport prediction, or the apparatus is to receive from the sender the viewport prediction, the viewport prediction indicating a change from the current viewing direction of the user to a new viewing direction of the user to happen after a lookahead time.

7. The apparatus of claim 1 , wherein, during the session of the 360° video communication, if a viewport size comprises a margin area, the apparatus is to receive an indication of a lens/distortion parameter used for rendering to assist the apparatus in cropping/warping a viewport.

8. An apparatus for a 360° video communication with a receiver, wherein

the apparatus is to

receive from the receiver an indication of a certain viewing direction of a 360° video at the receiver, and

transmit video data for the certain viewing direction of a 360° video to the receiver,

wherein, for the certain viewing direction of the 360° video, the apparatus is to receive a trigger for a switching during the session between

(i) first video data rendered by the apparatus and representing a 2D video at the certain viewing direction of the 360° video; and

(ii) second video data not rendered by the apparatus and representing at least a part of the 360° video to be transmitted by the apparatus using a certain projection.

9. The apparatus of claim 8 , wherein the apparatus is to,

in case the first video data is to be provided, render the video data, encode the rendered video data and transmit the encoded video data to the receiver, and

in case the second video data is to be provided, encode the video data using a certain projection, without rendering, encode one or more Supplementary Enhancement Information (SEI) messages indicating a projection type, a rotation and region-wise packing (RWP) constraints, and transmit the encoded video data and the encoded one or more messages to the receiver.

10. The apparatus of claim 8 , wherein the apparatus is to provide to the receiver the first video data or the second video data dependent on an end-to-end latency between the receiver and the sender.

11. The apparatus of claim 8 , wherein

at the beginning of a session of the 360° video communication, the apparatus is to negotiate with the receiver, and

when negotiating with the receiver, the apparatus is to send to the receiver Supplementary Enhancement Information (SEI) messages indicating one or more of a projection type, a rotation and region-wise packing (RWP) constraints.

12. The apparatus of claim 8 , wherein the apparatus comprises a viewport predictor providing a viewport prediction, or the apparatus is to receive from the receiver the viewport prediction, the viewport prediction indicating a change from the current viewing direction of the user of the receiver to a new viewing direction of the user to happen after a lookahead time.

13. The apparatus of claim 8 , wherein, during of a session of the 360° video communication, if a viewport size comprises the margin area, the apparatus is to send to the receiver an indication of a lens/distortion parameter used for rendering to assist the receiver in cropping/warping a viewport.

14. A 360° video communication system, comprising:

a sender comprising an apparatus of claim 8 , and

a receiver comprising an apparatus for a 360° video communication with a sender, wherein

the receiver is to

acquire from the sender video data dependent on a certain viewing direction of a 360° video, and

display the video data representing the certain viewing direction of the 360° video.

15. A method for a 360° video communication, the method comprising:

receiving, at a sender, an indication from a receiver of a certain viewing direction of a 360° video at the receiver, and

transmitting, by the sender, video data for the certain viewing direction of a 360° video to the receiver,

wherein, for the certain viewing direction of the 360° video, the method comprises triggering a switching during the session between

(i) first video data rendered by the sender and representing a 2D video at the certain viewing direction of the 360° video; and

(ii) second video data not rendered by the sender and representing at least a part of the 360° video to be transmitted by the sender using a certain projection.

16. The method of claim 15 , further comprising:

acquiring, by the receiver, the video data from the sender dependent on the certain viewing direction of a 360° video at the receiver, and

displaying, at the receiver the video data representing the certain viewing direction of the 360° video.

17. A non-transitory digital storage medium having a computer program stored thereon to perform, when said computer program is run by a computer, the method for a 360° video communication, the method comprising:

receiving, at a sender, an indication from a receiver of a certain viewing direction of a 360° video at the receiver, and

transmitting, by the sender, video data for the certain viewing direction of a 360° video to the receiver,

wherein, for the certain viewing direction of the 360° video, the method comprises triggering a switching during the session between

(i) first video data rendered by the sender and representing a 2D video at the certain viewing direction of the 360° video; and

(ii) second video data not rendered by the sender and representing at least a part of the 360° video to be transmitted by the sender using a certain projection.

Priority Claims (2)

Continuity (3)

References Cited (41)