Deep learning based voice extraction and primary-ambience decomposition for stereo to surround upmixing with dialog-enhanced center channel
One embodiment provides a computer-implemented method that includes determining directional sounds from a content mix using a machine learning unmixing model. The directional sounds are panned in an upmixed signal. Signal-dependent upmixing gains for specific frequency bins are computed on a frame-basis using a machine learning model for the upmixed signal. Dedicated voice clarity gains are computed using a hearing impairment model for multiple hearing-impaired profiles for achieving dialog enhancement. The signal dependent upmixing gains and voice clarity gains are transmitted as metadata with a downmixed signal representing the content mix.
1 . A computing method comprising:
determining directional sounds from a content mix using a machine learning unmixing model;
panning the directional sounds in an upmixed signal;
computing signal-dependent upmixing gains for specific frequency bins on a frame-basis using a machine learning model for the upmixed signal; and
computing dedicated voice clarity gains using a hearing impairment model for a plurality of hearing-impaired profiles for achieving dialog enhancement;
wherein the signal dependent upmixing gains and voice clarity gains are transmitted as metadata with a downmixed signal representing the content mix.
2 . The method of claim 1 , further comprising:
performing, by the computing device, a primary-ambience decomposition process for the upmixed signal.
3 . The method of claim 2 , further comprising:
applying the signal-dependent upmixing gains to downmixed signal components.
4 . The method of claim 2 , wherein the content mix comprises a voice content mix.
5 . The method of claim 2 , wherein during upmixing, the signal-dependent upmixing gains are applied to primary and ambient signals to generate a final output.
6 . The method of claim 2 , wherein the signal-dependent upmixing gains are embedded as audio-codec metadata.
7 . The method of claim 6 , wherein the audio-codec metadata is transmitted with encoded downmixed stereo signals.
8 . A non-transitory processor-readable medium that includes a program that when executed by a processor performs dialog enhancement of extracted sources of an unmixed signal, comprising:
determining, by the processor, directional sounds from a content mix using a machine learning unmixing model;
panning, by the processor, the directional sounds in an upmixed signal;
computing, by the processor, signal-dependent upmixing gains for specific frequency bins on a frame-basis using a machine learning model for the upmixed signal; and
computing, by the processor, dedicated voice clarity gains using a hearing impairment model for a plurality of hearing-impaired profiles for achieving dialog enhancement;
wherein the signal dependent upmixing gains and voice clarity gains are transmitted as metadata with a downmixed signal representing the content mix.
9 . The non-transitory processor-readable medium of claim 8 , further comprising:
performing, by the processor, a primary-ambience decomposition process for the upmixed signal.
10 . The non-transitory processor-readable medium of claim 9 , further comprising:
applying the signal-dependent upmixing gains to downmixed signal components.
11 . The non-transitory processor-readable medium of claim 9 , wherein the content mix comprises a voice content mix.
12 . The non-transitory processor-readable medium of claim 9 , wherein during upmixing, the signal-dependent upmixing gains are applied to primary and ambient signals to generate a final output.
13 . The non-transitory processor-readable medium of claim 9 , wherein the signal-dependent upmixing gains are embedded as audio-codec metadata.
14 . The non-transitory processor-readable medium of claim 13 , wherein the audio-codec metadata is transmitted with encoded downmixed stereo signals.
15 . An apparatus comprising:
a memory storing instructions; and
at least one processor executes the instructions including a process configured to:
determine directional sounds from a content mix using a machine learning unmixing model;
pan the directional sounds in an upmixed signal;
compute signal-dependent upmixing gains for specific frequency bins on a frame-basis using a machine learning model for the upmixed signal; and
compute dedicated voice clarity gains using a hearing impairment model for a plurality of hearing-impaired profiles for achieving dialog enhancement;
wherein the signal dependent upmixing gains and voice clarity gains are transmitted as metadata with a downmixed signal representing the content mix.
16 . The apparatus of claim 15 , further comprising:
performing, by the computing device, a primary-ambience decomposition process for the upmixed signal.
17 . The apparatus of claim 16 , further comprising:
applying the signal-dependent upmixing gains to downmixed signal components.
18 . The apparatus of claim 16 , wherein the content mix comprises a voice content mix.
19 . The apparatus of claim 16 , wherein during upmixing, the signal-dependent upmixing gains are applied to primary and ambient signals to generate a final output.
20 . The apparatus of claim 16 , wherein the signal-dependent upmixing gains are embedded as audio-codec metadata, and the audio-codec metadata is transmitted with encoded downmixed stereo signals.