Granted Patent B2

US 12,268,959 · App. 17/208,991 · Granted Apr 8, 2025

Inventors: Christof Fersch (Neumarkt, DE); Nicolas R. Tsingos (San Francisco, CA)

Assignees: DOLBY INTERNATIONAL AB; DOLBY LABORATORIES LICENSING CORPORATION

A63F13/428A63F13/213G06F3/012H04L67/131H04S3/008H04S7/303H04S2400/01H04S2400/11H04S2420/11

Abstract

Claims (68)

1. A system comprising a first apparatus for reproducing media content and a second apparatus storing the media content,

wherein the first apparatus is adapted to:

obtain pose information indicative of a position or orientation of a user;

transmit the pose information to the second apparatus;

receive rendered media content and predicted pose information from the second apparatus;

compare the predicted pose information to actual pose information, wherein the actual pose information is pose information obtained at a timing at which the rendered media content is actually processed by the first apparatus for reproduction;

update the rendered media content based on a result of the comparison;

the second apparatus is adapted to:

determine the predicted pose information based on the pose information received from the first apparatus and previous pose information received from the first apparatus at a previous timing;

render the media content based on the predicted pose information to obtain the rendered media content; and

transmit the rendered media content and the predicted pose information to the first apparatus for reproduction.

2. The system according to claim 1 , wherein the media content comprises audio content and the rendered media content comprises rendered audio content; and/or

the media content comprises video content and the rendered media content comprises rendered video content.

3. The system according to claim 1 , wherein the media content comprises audio content and the rendered media content comprises rendered audio content; and

the first apparatus is further adapted to generate an audible representation of the rendered audio content.

4. The system according to claim 2 , wherein the audio content is one of First Order Ambisonics, FOA-based, Higher Order Ambisonics, HOA-based, object-based, or channel based audio content, or a combination of two or more of FOA-based, HOA-based, object-based, or channel based audio content.

5. The system according to claim 2 , wherein the rendered audio content is one of binaural audio content, FOA audio content, HOA audio content, or channel-based audio content, or a combination of two or more of binaural audio content, FOA audio content, HOA audio content, or channel-based audio content.

6. The system according to claim 1 , wherein the predicted pose information is predicted for an estimate of a timing at which the rendered media content is expected to be processed by the first apparatus for reproduction.

7. The system according to claim 1 , wherein the rendered media content is transmitted to the first apparatus in uncompressed form.

8. The system according to claim 1 , wherein the second apparatus is further adapted to encode the rendered media content before transmission to the first apparatus; and

the first apparatus is further adapted to decode the encoded rendered media content after reception at the first apparatus.

9. The system according to claim 6 , wherein the estimate of the timing at which the rendered media content is expected to be processed by the first apparatus for reproduction includes an estimation of a time that is necessary for encoding and decoding the rendered media content and/or an estimate of a time that is necessary for transmitting the rendered media content to the first apparatus.

10. The system according to claim 1 , wherein the predicted pose information is obtained further based on an estimate of a time that is necessary for encoding and decoding the rendered media content and/or an estimate of a time that is necessary for transmitting the rendered media content to the first apparatus.

11. The system according to claim 1 , wherein the second apparatus is further adapted to:

determine gradient information indicative of how the rendered media content changes in response to changes of the pose information; and

transmit the gradient information to the first apparatus together with the rendered media content; and

the first apparatus is further adapted to:

update the rendered media content based on the gradient information and the result of the comparison.

12. The system according to claim 1 , wherein the media content comprises audio content and the rendered media content comprises rendered audio content;

the first apparatus is further adapted to transmit environmental information indicative of acoustic characteristics of an environment in which the first apparatus is located to the second apparatus; and

the rendering the media content is further based on the environmental information.

13. The system according to claim 1 , wherein the media content comprises audio content and the rendered media content comprises rendered audio content;

the first apparatus is further adapted to transmit morphologic information indicative of a morphology of the user or part of the user to the second apparatus; and

the rendering the media content is further based on the morphologic information.

14. A second apparatus for providing media content for reproduction by a first apparatus, the second apparatus adapted to:

receive pose information indicative of a position or orientation of a user of the first apparatus,

wherein the second apparatus comprises:

a position predictor configured to determine predicted pose information based on the pose information received from the first apparatus and previous pose information received from the first apparatus at a previous timing;

a renderer configured to render the media content based on the predicted pose information to obtain rendered media content; and

a transmission unit configured to transmit the rendered media content and the predicted pose information to the first apparatus for updating the rendered media content for reproduction based on a comparison of the predicted pose information with actual pose information determined at the first apparatus, wherein the actual pose information is pose information obtained at a timing at which the rendered media content is actually processed by the first apparatus for reproduction.

15. The second apparatus according to claim 14 , wherein the media content comprises audio content and the rendered media content comprises rendered audio content; and/or

the media content comprises video content and the rendered media content comprises rendered video content.

16. The second apparatus according to claim 15 , wherein the audio content is one of First Order Ambisonics, FOA-based, Higher Order Ambisonics, HOA-based, object-based, or channel based audio content, or a combination of two or more of FOA-based, HOA-based, object-based, or channel based audio content.

17. The second apparatus according to claim 15 , wherein the rendered audio content is one of binaural audio content, FOA audio content, HOA audio content, or channel-based audio content, or a combination of two or more of binaural audio content, FOA audio content, HOA audio content, or channel-based audio content.

18. The second apparatus according to claim 14 , wherein the predicted pose information is predicted for an estimate of a timing at which the rendered media content is expected to be processed by the first apparatus for reproduction.

19. The second apparatus according to claim 14 , wherein the rendered media content is transmitted to the first apparatus in uncompressed form.

20. The second apparatus according to claim 14 , further adapted to encode the rendered media content before transmission to the first apparatus.

21. The second apparatus according to claim 18 , wherein the estimate of the timing at which the rendered media content is expected to be processed by the first apparatus for reproduction includes an estimation of a time that is necessary for encoding and decoding the rendered media content and/or an estimate of a time that is necessary for transmitting the rendered media content to the first apparatus.

22. The second apparatus according to claim 14 , wherein the predicted pose information is obtained further based on an estimate of a time that is necessary for encoding and decoding the rendered media content and/or an estimate of a time that is necessary for transmitting the rendered media content to the first apparatus.

23. The second apparatus according to claim 14 , further adapted to:

determine gradient information indicative of how the rendered media content changes in response to changes of the pose information; and

transmit the gradient information to the first apparatus together with the rendered media content.

24. The second apparatus according to claim 14 , wherein the media content comprises audio content and the rendered media content comprises rendered audio content;

the second apparatus is further adapted to receive environmental information indicative of acoustic characteristics of an environment in which the first apparatus is located from the first apparatus; and

the rendering the media content is further based on the environmental information.

25. The second apparatus according to claim 14 , wherein the media content comprises audio content and the rendered media content comprises rendered audio content;

the second apparatus is further adapted to receive morphologic information indicative of a morphology of the user or part of the user from the first apparatus; and

the rendering the media content is further based on the morphologic information.

26. A first apparatus for reproducing media content provided by a second apparatus, the first apparatus adapted to:

obtain pose information indicative of a position and/or orientation of a user of the first apparatus;

transmit the pose information to the second apparatus;

receive rendered media content and predicted pose information from the second apparatus, wherein the rendered media content has been obtained by rendering the media content based on the predicted pose information;

compare the predicted pose information to an actual pose information, wherein the actual pose information is pose information obtained at a timing at which the rendered media content is actually processed by the first apparatus for reproduction;

update the rendered media content based on a result of the comparison; and

reproduce the updated rendered media content.

27. The system according to claim 1 , wherein update the rendered media content comprises extrapolating the rendered media content.

28. The system according to claim 27 , wherein extrapolating the rendered media content comprises applying blind upmixing to the rendered media content.

29. The system according to claim 27 , wherein extrapolating the rendered media content comprises applying a local rotation to the rendered media content.

Assignments (1)

Priority Claims (1)

Continuity (4)

References Cited (35)