Interpolation for decomposed representations of a sound field
In general, techniques are described for performing an interpolation with respect to decomposed versions of a sound field. A device comprising one or more processors may be configured to perform the techniques. The processors may be configured to obtain decomposed interpolated spherical harmonic coefficients for a time segment by, at least in part, performing an interpolation with respect to a first decomposition of a first plurality of spherical harmonic coefficients and a second decomposition of a second plurality of spherical harmonic coefficients.
1. A method comprising:
obtaining, by an audio decoding device, from a first frame of a bitstream representative of compressed audio data, data indicative of a first decomposition of a first portion of a first plurality of spherical harmonic coefficients;
obtaining, by the audio decoding device, from a second frame of the bitstream, data indicative of a second decomposition of a second portion of a second plurality of spherical harmonic coefficients, wherein the first decomposition of the first plurality of spherical harmonic coefficients and the second decomposition of the second plurality of spherical harmonic coefficients are each defined in the spherical harmonic domain and are each indicative of a shape and a width of a corresponding predominant signal present in a soundfield represented by the first and second plurality of spherical harmonic coefficients;
performing, by the audio decoding device, an interpolation with respect to the first decomposition and the second decomposition to obtain decomposed interpolated spherical harmonic coefficients;
obtaining, by the audio decoding device and from the bitstream, a predominant signal corresponding to the decomposed interpolated spherical harmonic coefficients;
rendering, by the audio decoding device, one or more speaker feeds based on the decomposed interpolated spherical harmonic coefficients and the corresponding predominant signal; and
outputting, by the audio decoding device, the one or more speaker feeds to one or more speakers.
2. The method of claim 1 , wherein the first decomposition comprises a first V vector.
3. The method of claim 1 , wherein the second decomposition comprises a second V vector.
4. The method of claim 1 , wherein performing the interpolation comprises performing the interpolation with respect to a first V vector and a second V vector to obtain an interpolated V vector corresponding to the predominant signal.
5. The method of claim 1 , wherein the time segment comprises a sub-frame of the first frame.
6. The method of claim 1 , wherein the time segment comprises a time sample of the first frame.
7. The method of claim 1 , further comprising:
receiving a first artificial time component and a second artificial time component;
and
applying inverses of the interpolated decompositions to the first artificial time component to recover a first time component and to the second artificial time component to recover a second time component.
8. The method of claim 1 , wherein obtaining the decomposed interpolated spherical harmonic coefficients for the time segment comprises interpolating a first spatial component of the first plurality of spherical harmonic coefficients and the second spatial component of the second plurality of spherical harmonic coefficients.
9. The method of claim 8 , wherein the first spatial component is representative of M time segments of spherical harmonic coefficients for the first plurality of spherical harmonic coefficients and the second spatial component is representative of M time segments of spherical harmonic coefficients for the second plurality of spherical harmonic coefficients.
10. The method of claim 8 ,
wherein the first spatial component is representative of M time segments of spherical harmonic coefficients for the first plurality of spherical harmonic coefficients and the second spatial component is representative of M time segments of spherical harmonic coefficients for the second plurality of spherical harmonic coefficients, and
wherein obtaining the decomposed interpolated spherical harmonic coefficients for the time segment comprises interpolating the last N elements of the first spatial component and the first N elements of the second spatial component.
11. The method of claim 1 , wherein the second plurality of spherical harmonic coefficients are subsequent to the first plurality of spherical harmonic coefficients in the time domain.
12. The method of claim 1 , wherein the first and second plurality of spherical harmonic coefficients each represent a planar wave representation of the sound field.
13. The method of claim 1 , wherein the first and second plurality of spherical harmonic coefficients each represent one or more mono-audio objects mixed together.
14. The method of claim 1 , wherein the first and second plurality of spherical harmonic coefficients each comprise respective first and second spherical harmonic coefficients that represent a three dimensional sound field.
15. The method of claim 1 , wherein the first and second plurality of spherical harmonic coefficients are each associated with at least one spherical basis function having an order greater than one.
16. The method of claim 1 , wherein the first and second plurality of spherical harmonic coefficients are each associated with at least one spherical basis function having an order equal to four.
17. The method of claim 1 , wherein the interpolation is a weighted interpolation of the first decomposition and second decomposition, wherein weights of the weighted interpolation applied to the first decomposition are inversely proportional to a time represented by vectors of the first and second decomposition and wherein weights of the weighted interpolation applied to the second decomposition are proportional to a time represented by vectors of the first and second decomposition.
18. The method of claim 1 , wherein the decomposed interpolated spherical harmonic coefficients smooth at least one of spatial components and time components of the first plurality of spherical harmonic coefficients and the second plurality of spherical harmonic coefficients.
19. The method of claim 1 , further comprising:
obtaining the bitstream that includes:
(1) a representation of the decomposed interpolated spherical harmonic coefficients for the time segment; and
(2) an indication of a type of the interpolation.
20. The method of claim 19 , wherein the indication comprises one or more bits that map to the type of interpolation.
21. The method of claim 1 , further comprising reproducing, by the one or more speakers and based on the speaker feeds, a soundfield represented by the interpolated decomposed spherical harmonic coefficients.
22. The method of claim 1 , further comprising reconstructing, by the audio decoding device and based on the decomposed interpolated spherical harmonic coefficients and the predominant signal, the spherical harmonic coefficients,
wherein rendering the one or more speaker feeds comprises rendering, based on the reconstructed spherical harmonic coefficients, the one or more speaker feeds.
23. The method of claim 1 ,
wherein rendering the one or more speaker feeds comprises rendering, based on the decomposed interpolated spherical harmonic coefficients, one or more loudspeaker feeds, and
wherein the one or more speakers comprise one or more loudspeakers.
24. The method of claim 1 ,
wherein rendering the one or more speaker feeds comprises rendering, based on the decomposed interpolated spherical harmonic coefficients, one or more binaural audio headphone feeds, and
wherein the one or more speakers comprise one or more headphone speakers.
25. The method of claim 1 , further comprising:
performing dequantization with respect to the data indicative of the first decomposition to obtain the first decomposition; and
performing dequantization with respect to the data indicative of the second decomposition to obtain the second decomposition.
26. A device comprising:
one or more processors configured to;
obtain, from a first frame of a bitstream representative of compressed audio data, data indicative of a first decomposition of a first portion of a first plurality of spherical harmonic coefficients;
obtain, from a second frame of the bitstream, data indicative of a second decomposition of a second portion of a second plurality of spherical harmonic coefficients, wherein the first decomposition of the first plurality of spherical harmonic coefficients and the second decomposition of the second plurality of spherical harmonic coefficients are each defined in the spherical harmonic domain and are each indicative of a shape and a width of a corresponding predominant signal present in a soundfield represented by the first and second plurality of spherical harmonic coefficients;
perform an interpolation with respect to the first decomposition and the second decomposition;
obtain, from the bitstream, a predominant signal corresponding to the decomposed interpolated spherical harmonic coefficients;
render one or more speaker feeds based on the decomposed interpolated spherical harmonic coefficients and the corresponding predominant signal; and
output the one or more speaker feeds to one or more speakers; and
a memory coupled to the one or more processors, and configured to store the speaker feeds.
27. The device of claim 26 , wherein the first decomposition comprises a first V vector.
28. The device of claim 26 , wherein the second decomposition comprises a second V vector.
29. The device of claim 26 ,
wherein the one or more processors are configured to perform the interpolation with respect to a first V matrix and a second V matrix to obtain an interpolated V vector corresponding to the predominant signal.
30. The device of claim 26 , wherein the time segment comprises a time sample of the first frame.
31. The device of claim 26 , wherein the one or more processors are further configured to:
receive a first artificial time component and a second artificial time component; and
apply inverses of the interpolated decompositions to the first artificial time component to recover the first time component and to the second artificial time component to recover the second time component.
32. The device of claim 26 , wherein the one or more processors are configured to interpolate a first spatial component of the first plurality of spherical harmonic coefficients and the second spatial component of the second plurality of spherical harmonic coefficients.
33. The device of claim 32 , wherein the first spatial component comprises a first U matrix representative of left-singular vectors of the first plurality of spherical harmonic coefficients.
34. The device of claim 32 , wherein the second spatial component comprises a second U matrix representative of left-singular vectors of the second plurality of spherical harmonic coefficients.
35. The device of claim 32 , wherein the first spatial component is representative of M time segments of spherical harmonic coefficients for the first plurality of spherical harmonic coefficients and the second spatial component is representative of M time segments of spherical harmonic coefficients for the second plurality of spherical harmonic coefficients.
36. The device of claim 32 ,
wherein the first spatial component is representative of M time segments of spherical harmonic coefficients for the first plurality of spherical harmonic coefficients and the second spatial component is representative of M time segments of spherical harmonic coefficients for the second plurality of spherical harmonic coefficients, and
wherein the one or more processors are configured to interpolate the last N elements of the first spatial component and the first N elements of the second spatial component.
37. The device of claim 26 , wherein the second plurality of spherical harmonic coefficients are subsequent to the first plurality of spherical harmonic coefficients in the time domain.
38. The device of claim 26 , wherein the first and second plurality of spherical harmonic coefficients each represent a planar wave representation of the sound field.
39. The device of claim 26 , wherein the first and second plurality of spherical harmonic coefficients each represent one or more mono-audio objects mixed together.
40. The device of claim 26 , wherein the first and second plurality of spherical harmonic coefficients are each associated with at least one spherical basis function having an order greater than one.
41. The device of claim 26 , wherein the first and second plurality of spherical harmonic coefficients are each associated with at least one spherical basis function having an order equal to four.
42. The device of claim 26 , wherein the one or more processors are further configured to obtain the bitstream that includes a representation of the decomposed interpolated spherical harmonic coefficients for the time segment, and an indication of a type of the interpolation.
43. The device of claim 42 , wherein the indication comprises one or more bits that map to the type of interpolation.
44. The device of claim 26 , further comprising the one or more speakers, configured to reproduce, based on the speaker feeds, a soundfield represented by the interpolated decomposed spherical harmonic coefficients.
45. The device of claim 26 ,
wherein the one or more processors are further configured to reconstruct, based on the decomposed interpolated spherical harmonic coefficients and the predominant signal, the spherical harmonic coefficients,
wherein the one or more processors are configured to render, based on the reconstructed spherical harmonic coefficients, the one or more speaker feeds.
46. The device of claim 26 ,
wherein the one or more processors are configured to render, based on the decomposed interpolated spherical harmonic coefficients, one or more loudspeaker feeds, and
wherein the one or more speakers comprise one or more loudspeakers.
47. The device of claim 26 ,
wherein the one or more processors are configured to render, based on the decomposed interpolated spherical harmonic coefficients, one or more binaural audio headphone feeds, and
wherein the one or more speakers comprise one or more headphone speakers.
48. The device of claim 26 , wherein the one or more processors are further configured to:
perform dequantization with respect to the data indicative of the first decomposition to obtain the first decomposition; and
perform dequantization with respect to the data indicative of the second decomposition to obtain the second decomposition.
49. A device comprising:
means for obtaining, by an audio decoding device, from a first frame of a bitstream representative of compressed audio data, a first decomposition of a first portion of a first plurality of spherical harmonic coefficients;
means for obtaining, by the audio decoding device, from a second frame of the bitstream, a second decomposition of a second portion of a second plurality of spherical harmonic coefficients, wherein the first decomposition of the first plurality of spherical harmonic coefficients and the second decomposition of the second plurality of spherical harmonic coefficients are each defined in the spherical harmonic domain and are each indicative of a shape and a width of a corresponding predominant signal present in a soundfield represented by the first and second plurality of spherical harmonic coefficients;
means for performing an interpolation with respect to the first decomposition and the second decomposition;
means for obtaining, from the bitstream, a predominant signal corresponding to the decomposed interpolated spherical harmonic coefficients;
means for rendering one or more speaker feeds based on the decomposed interpolated spherical harmonic coefficients and the corresponding predominant signal; and
means for outputting the one or more speaker feeds to one or more speakers.
50. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to:
obtain, from a first frame of a bitstream representative of compressed audio data, a first decomposition of a first portion of a first plurality of spherical harmonic coefficients;
obtain, from a second frame of the bitstream, a second decomposition of a second portion of a second plurality of spherical harmonic coefficients, wherein the first decomposition of the first plurality of spherical harmonic coefficients and the second decomposition of the second plurality of spherical harmonic coefficients are each defined in the spherical harmonic domain and are each indicative of a shape and a width of a corresponding predominant signal present in a soundfield represented by the first and second plurality of spherical harmonic coefficients;
perform an interpolation with respect to the first decomposition and the second decomposition;
obtain, from the bitstream, a predominant signal corresponding to the decomposed interpolated spherical harmonic coefficients;
render one or more speaker feeds based on the decomposed interpolated spherical harmonic coefficients and the corresponding predominant signal; and
output the one or more speaker feeds to one or more speakers.