Hierarchical spatial resolution codec
Disclosed is a hierarchical spatial resolution codec that adaptively adjusts the representations of immersive audio content as the target bandwidth for delivering the audio content changes. The audio content may be represented by an adaptive number of content types such as channels/objects, higher-order ambisonics (HOA), and encoded by adaptive spatial coding techniques to support the target bitrate of a transmission channel or user. Adaptive spatial coding techniques may include adaptive channel/object spatial encoding techniques to generate an adaptive number of channels/objects, and adaptive HOA spatial encoding or HOA compression techniques to generate an adaptive order of the HOA. The adaptation may be a function of the target bitrate that is associated with a desired quality, and an analysis that determines the priority of the channels, objects, and HOA. High priority channels/objects may be encoded into a high quality bit-stream while low priority channels/objects may be converted and encoded as HOA.
1. A method of encoding audio content, the method comprising:
receiving, by an encoding device, the audio content, the audio content being represented by one or more content types, a first content type including a plurality of scene elements;
determining priorities of the plurality of scene elements of the first content type;
encoding an adaptive number of the plurality of scene elements of the first content type into a first content stream based on the priorities of the plurality of scene elements and a target bitrate for transmitting the audio content;
encoding into a second content stream remaining scene elements of the first content type not selected for encoding into the first content stream, the second content stream representing encoding of a second content type;
wherein as the target bit rate changes, the adaptive number of scene elements of the first content type is selected based on selecting scene elements having higher priorities than the priorities of the remaining scene elements; and
generating a transport stream that includes the first content stream and the second content stream for transmission based on the target bitrate.
2. The method of claim 1 , wherein the first content type has a higher quality of sound field representation of the audio content than the second content type.
3. The method of claim 1 , wherein a bit-rate for supporting a transmission of the first content type is higher than a bit-rate for supporting a transmission of the second content type.
4. The method of claim 1 , wherein determining the priorities of the plurality of scene elements of the first content type comprises:
generating a priority ranking of the plurality of scene elements of the first content type based on a spatial saliency of the plurality of scene elements, wherein a scene element having a higher spatial saliency has a higher quality of sound field representation than another scene element having a lower spatial saliency.
5. The method of claim 1 , wherein encoding into the second content stream, based on the target bitrate and priorities of scene elements of the second content type, the remaining scene elements of the first content type not selected for encoding into the first content stream comprises:
converting the remaining scene elements of the first content type into scene elements of the second content type; and
encoding the converted scene elements combined with scene elements of the second content type received from the audio content to generate the second content stream based on the target bitrate.
6. The method of claim 5 , wherein encoding the converted scene elements combined with scene elements of the second content type received from the audio content comprises:
determining priorities of a plurality of scene elements of the second content type that includes the converted scene elements and the scene elements of the second content type received from the audio content;
encoding an adaptive number of the plurality of scene elements of the second content type into the second content stream based on the priorities of the plurality of scene elements of the second content type and the target bitrate;
encoding into a third content stream based on the target bitrate remaining scene elements of the second content type not selected for encoding into the second content stream, the third content stream representing encoding of a third content type; and
generating the transport stream to include the third content stream.
7. The method of claim 6 , wherein the first content type has a higher quality of sound field representation of the audio content than the second content type and the second content type has a higher quality of sound field representation of the audio content than the third content type.
8. The method of claim 6 , wherein a bit-rate for supporting a transmission of the first content type is higher than a bit-rate for supporting a transmission of the second content type, and the bit-rate for supporting a transmission of the second content type is higher than a bit-rate for supporting a transmission of the third content type.
9. The method of claim 5 , wherein determining the priorities of the plurality of scene elements of the second content type comprises:
generating a priority ranking of the plurality of scene elements of the second content type based on a spatial saliency of the plurality of scene elements, wherein a scene element having a higher spatial saliency has a higher quality of sound field representation than another scene element having a lower spatial saliency.
10. The method of claim 5 , wherein encoding the adaptive number of the plurality of scene elements of the second content type into the second content stream comprises:
selecting the adaptive number of the scene elements of the second content type based on the selected scene elements having higher priorities than the priorities of the remaining scene elements of the second content type not selected for encoding into the second content stream as the target bitrate changes.
11. The method of claim 1 , wherein encoding into the second content stream based on the target bitrate the remaining scene elements of the first content type not selected for encoding into the first content stream comprises:
converting a first subset of the remaining scene elements of the first content type into scene elements of the second content type;
encoding the converted scene elements into the second content stream based on the target bitrate;
encoding into a third content stream, based on the target bitrate, a second subset of the remaining scene elements of the first content type not converted into scene elements of the second type, the third content stream representing encoding of a third content type; and
generating the transport stream to include the third content stream.
12. The method of claim 1 , wherein generating the transport stream comprises:
performing baseline encoding and spatial encoding of the first content stream and the second content stream based on the target bitrate.
13. The method of claim 1 , wherein the audio content comprises voice dialogue as one of the content types, wherein the method further comprises:
encoding the voice dialogue into a speech stream based on the target bitrate; and
generating the transport stream to include the speech stream.
14. The method of claim 1 , wherein the first content type is associated with metadata that describe properties of the plurality of scene elements of the first content type,
wherein encoding the adaptive number of the plurality of scene elements of the first content type into the first content stream comprises:
encoding the metadata associated with the adaptive number of the plurality of scene elements into metadata of the first content stream based on the target bitrate,
wherein encoding into the second content stream based on the target bitrate the remaining scene elements of the first content type comprises:
encoding the metadata associated with the remaining scene elements into metadata of the second content stream based on the target bitrate,
and wherein generating the transport stream comprises:
combining the metadata of the first content stream and the metadata of the second content stream into one metadata transport stream based on the target bitrate.
15. The method of claim 14 , wherein the metadata associated with the first content type comprises metadata to aid the encoding device in determining the priorities of the plurality of scene elements of the first content type and to aid a decoding device in spatial decoding and rendering of the plurality of scene elements of the first content type.
16. The method of claim 1 , wherein encoding the adaptive number of the plurality of scene elements of the first content type into the first content stream comprises:
generating a plurality of candidate first content streams based on the priorities of the plurality of the scene elements and a plurality of target bitrates, the plurality of candidate first content streams encoding an adaptive number of the scene elements of the first content type,
wherein encoding into the second content stream based on the target bitrate the remaining scene elements of the first content type not selected for encoding into the first content stream comprises:
generating a plurality of candidate second content streams based on the plurality of target bitrates, the plurality of candidate second content streams encoding an adaptive number of scene elements of the second content type that includes the remaining scene elements of the first content type converted into scene elements of the second content type combined with scene elements of the second content type received from the audio content,
and wherein generating the transport stream comprises:
selecting one of the plurality of candidate first content streams and one of the plurality of candidate second content streams for the transport stream based on the target bitrate of a user.
17. The method of claim 16 , further comprising:
storing in a file the plurality of candidate first content streams and the plurality of candidate second content streams,
and wherein generating the transport stream comprises:
selecting from the file one of the plurality of candidate first content streams and one of the plurality of candidate second content streams for the transport stream based on the target bitrate of a user.
18. The method of claim 1 , wherein encoding the adaptive number of the plurality of scene elements of the first content type into the first content stream comprises:
generating the first content stream to encode an adaptive number of the scene elements of the first content type based on the priorities of the plurality of the scene elements and as the target bitrate of a user changes;
and wherein encoding into the second content stream based on the target bitrate the remaining scene elements of the first content type not selected for encoding into the first content stream comprises:
generating the second content stream to encode, as the target bitrate of the user changes, an adaptive number of scene elements of the second content type that includes the remaining scene elements of the first content type converted into scene elements of the second content type combined with scene elements of the second content type received from the audio content.
19. The method of claim 1 , wherein the first content type comprises audio channels or audio objects, wherein the plurality of scene elements of the first content type comprise a plurality of audio channels or a plurality of audio objects, and wherein the second content type comprises higher-order ambisonics (HOA).