Acoustic echo cancellation based sub band domain active speaker detection for audio and video conferencing applications

1. A method for detecting an active speaker in at least a two-way conference comprising:

analyzing real time audio in one or more sub band domains according to an echo canceller model, wherein the echo canceller model includes at least in part processing the real time audio using an acoustic echo cancellation linear filter;

determining, based on the analyzed real time audio, one or more audio metrics;

weighting, via a trained machine learning model, the one or more audio metrics based on importance of the one or more audio metrics for active speaker determination in the one or more sub band domains;

summing the one or more weighted audio metrics;

comparing the one or more summed weighted audio metrics and a hysteresis threshold;

in response to the one or more summed weighted audio metrics being greater than the hysteresis threshold, determining a speaker status as active; and

in response to the speaker status being active, removing one or more of residual echo or noise from the real time audio based on the weighted one or more audio metrics.

2. The method of claim 1 , wherein the one or more of residual echo or noise from the real time audio is removed nonlinearly based on the one or more audio metrics.

3. The method of claim 1 , wherein the one or more audio metrics determines the speed of the hysteresis model.

4. The method of claim 1 , wherein one or more weights used in weighting the one or more audio metrics is generated dynamically from a machine learned model.

5. The method of claim 1 , wherein the one or more audio metrics include cross correlations among two or more of echo, microphone data, linear acoustic echo canceller output per sub band, background noise state, echo state, voice activity, and pitch state.

6. The method of claim 1 , wherein the echo canceller model cancels echo linearly without distorting speech.

7. The method of claim 1 , wherein the real time audio is captured by a microphone proximate to a participant in the two-way conference.

8. A system for detecting an active speaker in at least a two-way conference comprising:

at least one receiver for receiving real time audio;

a conference server communicatively coupled to the at least one receiver, wherein the conference server is configured to:

analyze the real time audio in one or more sub band domains according to an echo canceller model, wherein the echo canceller model includes at least in part processing the real time audio using an acoustic echo cancellation linear filter;

determine, based on the analyzed real time audio, one or more audio metrics;

weight, via a trained machine learning model, the one or more audio metrics based on importance of the one or more audio metrics for active speaker determination in the one or more sub band domains;

sum the one or more weighted audio metrics;

compare the one or more summed weighted audio metrics and a hysteresis threshold;

in response to the one or more summed weighted audio metrics being greater than the hysteresis threshold, determine a speaker status as active; and

in response to the speaker status being active, remove one or more of residual echo or noise from the real time audio based on the weighted one or more audio metrics; and

a loudspeaker in communication with the conference server configured to output the real time audio.

9. The system of claim 8 , wherein the one or more of residual echo or noise from the real time audio is removed nonlinearly based on the one or more audio metrics.

10. The system of claim 8 , wherein the one or more audio metrics determines the speed of the hysteresis model.

11. The system of claim 8 , wherein one or more weights used in weighting the one or more audio metrics is generated dynamically from a machine learned model.

12. The system of claim 8 , wherein the one or more audio metrics include cross correlations among two or more of echo, microphone data, linear acoustic echo canceller output per sub band, background noise state, echo state, voice activity, and pitch state.

13. The system of claim 8 , wherein the echo canceller model cancels echo linearly without distorting speech.

14. A non-transitory computer-readable medium containing instructions that, when executed by a computing system, cause the computing system to:

analyze real time audio in one or more sub band domains according to an echo canceller model, wherein the echo canceller model includes at least in part processing the real time audio using an acoustic echo cancellation linear filter;

determine, based on the analyzed real time audio, one or more audio metrics;

weight, via a trained machine learning model, the one or more audio metrics based on importance of the one or more audio metrics for active speaker determination in the one or more sub band domains;

sum the one or more weighted audio metrics;

compare the one or more summed weighted audio metrics and a hysteresis threshold;

in response to the one or more summed weighted audio metrics being greater than the hysteresis threshold, determine a speaker status is active; and

in response to the speaker status being active, remove one or more of residual echo or noise from the real time audio based on the weighted one or more audio metrics.

15. The non-transitory computer-readable medium of claim 14 , wherein the one or more of residual echo or noise from the real time audio is removed nonlinearly based on the one or more audio metrics.

16. The non-transitory computer-readable medium of claim 14 , wherein the one or more audio metrics determines the speed of the hysteresis model.

17. The non-transitory computer-readable medium of claim 14 , wherein one or more weights used in weighting the one or more audio metrics is generated dynamically from a machine learned model.

18. The non-transitory computer-readable medium of claim 14 , wherein the one or more audio metrics include cross correlations among two or more of echo, microphone data, linear acoustic echo canceller output per sub band, background noise state, echo state, voice activity, and pitch state.

19. The non-transitory computer-readable medium of claim 14 , wherein the echo canceller model cancels echo linearly without distorting speech.