IP Library › Granted Patent US 11,606,661
Granted Patent B2
US 11,606,661 · App. 16/648,324 · Granted Mar 14, 2023

Recording and rendering spatial audio signals

Inventors: Juha Vilkamo (Helsinki, FI); Mikko-Ville Laitinen (Helsinki, FI)
Assignee: Nokia Technologies Oy
H04S7/302G06F3/011G06F3/017G06F3/16H04R3/005H04R5/027H04R5/04H04S2400/01H04S2400/11H04S2420/11
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,606,661
App. No.
16/648,324
Granted
Mar 14, 2023
Kind
B2
Abstract

Examples of the disclosure relate to a method, apparatus and computer program, the method including: obtaining audio signals wherein the audio signals represent spatial sound and can be used to render spatial audio using linear methods; obtaining spatial metadata corresponding to the spatial sound represented by the audio signals; and associating the spatial metadata with the obtained audio signals so that in a first rendering context the obtained audio signals can be rendered without using the spatial metadata and in a second rendering context the obtained audio signals can be rendered with using the spatial metadata.

Claims (69)

1. A method comprising:

obtaining audio signals wherein the audio signals represent spatial sound of a sound space and are configured to be used to render spatial audio using linear methods;

obtaining spatial metadata corresponding to the sound space represented with the audio signals, wherein the spatial metadata comprises metadata associated with respective frequency bands of the audio signals; and

associating the spatial metadata with the obtained audio signals so that in a first rendering context the obtained audio signals are used to render spatial audio that is determined using at least one of the linear methods and without using the spatial metadata, and in a second rendering context the obtained audio signals are used to render spatial audio using the spatial metadata, wherein using the spatial metadata to process the obtained audio signals comprises, at least:

using the spatial metadata in response to a movement of a listener to process audio signals that are determined using one or more of the linear methods, and

determining at least one direct part of the obtained audio signals and at least one ambient part of the obtained audio signals based, at least partially, on the spatial metadata,

wherein associating the spatial metadata with the obtained audio signals comprises at least one of:

transmitting the spatial metadata and the obtained audio signals to a rendering device; or

storing the spatial metadata with the obtained audio signals.

2. A method as claimed in claim 1 , wherein the spatial metadata comprises information relating to spatial properties of the sound space represented with the obtained audio signals.

3. A method as claimed in claim 1 , wherein the obtained audio signals are configured to at least one of:

use spherical harmonics to enable spatial information to be rendered;

comprise Ambisonic signals; or

be obtained from a plurality of spatially separated microphones.

4. A method as claimed in claim 1 , wherein the first or second rendering context comprises one or more of: a type of rendering device being used, the movement of the listener, or a spatial accuracy required during rendering.

5. A method as claimed in claim 1 , wherein the spatial metadata is configured to enable processing of the obtained audio signals so as to adjust the rendered audio in response to the movement of the listener, wherein the movement of the listener comprises at least one of: an at least partially rotational movement of the listener, or a nonrotational movement of the listener.

6. A method as claimed in claim 1 , wherein the obtained audio signals comprise first order Ambisonics signals, wherein the second rendering context is configured to enable the listener to experience the rendered spatial audio with spatial accuracy while moving with six degrees of freedom in the sound space, wherein the movement of the listener comprises a translational movement of the listener, wherein using the spatial metadata to process the obtained audio signals in the second rendering context is configured to provide, to the listener, a perception of the sound space associated with a location of the sound space that is determined based, at least partially, on the translational movement of the listener.

7. A method as claimed in claim 1 , wherein using the spatial metadata to process the obtained audio signals comprises using the spatial metadata in response to a movement of the listener to increase spatial accuracy of the rendered audio signals in relation to the movement of the listener.

8. A method comprising:

obtaining audio signals wherein the audio signals represent spatial sound of a sound space and are configured to be used to render spatial audio using linear methods;

obtaining spatial metadata corresponding to the sound space represented with the audio signals, wherein the spatial metadata comprises metadata associated with respective frequency bands of the audio signals, wherein the spatial metadata comprises at least one of: distance parameters, direction parameters, or energy ratio parameters; and

associating the spatial metadata with the obtained audio signals so that in a first rendering context the obtained audio signals are used to render spatial audio that is determined using at least one of the linear methods and without using the spatial metadata, and in a second rendering context the obtained audio signals are used to render spatial audio using the spatial metadata, wherein using the spatial metadata to process the obtained audio signals comprises using the spatial metadata in response to a movement of a listener to process audio signals that are determined using one or more of the linear methods, wherein associating the spatial metadata with the obtained audio signals comprises at least one of:

transmitting the spatial metadata and the obtained audio signals to a rendering device; or

storing the spatial metadata with the obtained audio signals.

9. A method comprising:

obtaining audio signals wherein the audio signals represent a sound scene and are configured to be used to render spatial audio using linear methods;

obtaining spatial metadata corresponding to the sound scene represented with the audio signals; and

associating the spatial metadata with the obtained audio signals so that in a first rendering context the obtained audio signals are used to render spatial audio without using the spatial metadata, and in a second rendering context the obtained audio signals are used to render spatial audio using the spatial metadata, wherein using the spatial metadata to process the obtained audio signals increases the spatial accuracy of the rendered audio signals, wherein associating the spatial metadata with the obtained audio signals comprises at least one of:

transmitting the spatial metadata and the obtained audio signals to a rendering device; or

storing the spatial metadata with the obtained audio signals,

wherein the spatial metadata comprises information indicating how energy levels in one or more frequency sub-bands of the obtained audio signals have been modified.

10. An apparatus comprising:

processing circuitry; and

memory circuitry including computer program code, the memory circuitry and the computer program code configured to, with the processing circuitry, enable the apparatus to:

obtain audio signals wherein the audio signals represent spatial sound of a sound space and are configured to be used to render spatial audio using linear methods;

obtain spatial metadata corresponding to the sound space represented with the audio signals, wherein the spatial metadata comprises metadata associated with respective frequency bands of the audio signals; and

associate the spatial metadata with the obtained audio signals so that in a first rendering context the obtained audio signals are used to render spatial audio that is determined using at least one of the linear methods and without using the spatial metadata, and in a second rendering context the obtained audio signals are used to render spatial audio using the spatial metadata, wherein using the spatial metadata to process the obtained audio signals comprises enabling the apparatus to, at least:

use the spatial metadata in response to a movement of a listener to process audio signals that are determined using one or more of the linear methods, and

determine at least one direct part of the obtained audio signals and at least one ambient part of the obtained audio signals based, at least partially, on the spatial metadata,

wherein associating the spatial metadata with the obtained audio signals comprises enabling the apparatus to at least one of:

transmit the spatial metadata and the obtained audio signals to a rendering device; or

store the spatial metadata with the obtained audio signals.

11. An apparatus as claimed in claim 10 , wherein the audio signals are configured to be used to render spatial audio in response to at least partially rotational movement of the listener, wherein rendering spatial audio without using the spatial metadata comprises rendering the audio signals using at least a first one of the linear methods, wherein using the spatial metadata to process the obtained audio signals comprises using the spatial metadata in response to, at least, translational movement of the listener.

12. An apparatus as claimed in claim 10 , wherein the spatial metadata comprises information relating to spatial properties of the sound space represented with the obtained audio signals.

13. An apparatus as claimed in claim 10 , wherein the obtained audio signals are configured to at least one of:

use spherical harmonics to enable spatial information to be rendered;

comprise Ambisonic signals; or

be obtained from a plurality of spatially separated microphones.

14. An apparatus as claimed in claim 10 , wherein the first or second rendering context comprises one or more of: a type of rendering device being used, the movement of the listener, or a spatial accuracy required during rendering.

15. An apparatus as claimed in claim 10 , wherein the spatial metadata is configured to enable processing of the obtained audio signals so as to adjust the rendered audio in response to the movement of the listener.

16. An apparatus as claimed in claim 10 , wherein the obtained audio signals comprise first order Ambisonics signals.

17. An apparatus as claimed in claim 10 , wherein using the spatial metadata to process the obtained audio signals comprises using the spatial metadata in response to a movement of the listener to increase spatial accuracy of the rendered audio signals in relation to the movement of the listener.

18. An apparatus comprising:

processing circuitry; and

memory circuitry including computer program code, the memory circuitry and the computer program code configured to, with the processing circuitry, enable the apparatus to:

obtain audio signals wherein the audio signals represent spatial sound of a sound space and are configured to be used to render spatial audio using linear methods;

obtain spatial metadata corresponding to the sound space represented with the audio signals, wherein the spatial metadata comprises metadata associated with respective frequency bands of the audio signals, wherein the spatial metadata comprises at least one of: distance parameters, direction parameters, or energy ratio parameters; and

associate the spatial metadata with the obtained audio signals so that in a first rendering context the obtained audio signals are used to render spatial audio that is determined using at least one of the linear methods and without using the spatial metadata, and in a second rendering context the obtained audio signals are used to render spatial audio using the spatial metadata, wherein using the spatial metadata to process the obtained audio signals comprises using the spatial metadata in response to a movement of a listener to process audio signals that are determined using one or more of the linear methods, wherein associating the spatial metadata with the obtained audio signals comprises enabling the apparatus to at least one of:

transmit the spatial metadata and the obtained audio signals to a rendering device; or

store the spatial metadata with the obtained audio signals.

19. An apparatus comprising:

processing circuitry; and

memory circuitry including computer program code, the memory circuitry and the computer program code configured to, with the processing circuitry, enable the apparatus to:

obtain audio signals wherein the audio signals represent a sound scene and are configured to be used to render spatial audio using linear methods;

obtain spatial metadata corresponding to the sound scene represented with the audio signals; and

associate the spatial metadata with the obtained audio signals so that in a first rendering context the obtained audio signals are used to render spatial audio without using the spatial metadata, and in a second rendering context the obtained audio signals are used to render spatial audio using the spatial metadata, wherein using the spatial metadata to process the obtained audio signals increases the spatial accuracy of the rendered audio signals, wherein associating the spatial metadata with the obtained audio signals comprises enabling the apparatus to at least one of:

transmit the spatial metadata and the obtained audio signals to a rendering device; or

store the spatial metadata with the obtained audio signals,

wherein the spatial metadata comprises information indicating how energy levels in one or more frequency sub-bands of the obtained audio signals have been modified.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 20, 2020
From: VILKAMO, JUHA TAPIO; LAITINEN, MIKKO-VILLE ILARI
To: NOKIA TECHNOLOGIES OY
Reel/Frame 052174/0327 →
Priority Claims (1)
GB 1715863 · Sep 29, 2017 · national
Continuity (1)
Related Publication 20200260206A1 · Aug 13, 2020
Cited By (1)
US 12,659,684