Context inclusive interruption classification for dialog driven systems

1 . A system, comprising:

at least one processor; and

a memory, storing program instructions that when executed by the at least one processor, cause the at least on processor to:

during an audio presentation of dialog generated by the dialog driven system to a participant:

detecting voice audio;

responsive to detecting the voice audio, initiate a post-processing stage for the detected voice audio, the post-processing stage comprising:

generation of an acoustic encoding of the voice audio using a pre-trained audio encoder machine learning model created with self-supervised fine-tuning, wherein the acoustic encoding comprises one or more acoustic features;

generation of one or more context inclusive encodings for the voice audio comprising a text encoding of dialog not yet presented before the detected voice audio; and

application of an interruption classifier machine learning model that classifies the voice audio based, at least in part, on the acoustic encoding, including as inputs to the interruption classifier machine learning model the one or more acoustic features, and the one or more context inclusive encodings, including the text encoding of dialog not yet presented before the detected voice audio, wherein the voice audio is classified as a false interruption of the dialog; and

ignore the voice audio and continue the audio presentation of the dialog responsive to the false interruption classification of the voice audio.

2 . The system of claim 1 , wherein the memory stores further program instructions that when executed further cause the at least one processor to:

during an audio presentation of a second dialog generated by the dialog driven system to a second participant, detecting a second voice audio;

responsive to detecting the second voice audio, initiate the post-processing stage for the detected second voice audio, the post-processing stage comprising:

generation of a second acoustic encoding of the second voice audio using the pre-trained audio encoder machine learning model created with self-supervised fine-tuning; and

application of the machine learning model that classifies the second voice audio based, at least in part, on the second acoustic encoding, wherein the second voice audio is classified as a true interruption of the second dialog; and

stop the audio presentation of the second dialog responsive to the true interruption classification of the second voice audio to accept the second voice audio as a response to the second dialog.

3 . The system of claim 1 , wherein the acoustic encoding is combined with the one or more other context inclusion encodings performed as part of the post-processing stage and provided as combined input to the interruption classifier machine learning model that classifies the voice audio.

4 . The system of claim 1 , wherein the dialog driven system is an application hosted as part of a dialog driven application management service and wherein the dialog is generated as part of an interaction sequence specified for the application via an interface of the dialog driven application management service.

5 . A method, comprising:

during an audio presentation of dialog generated by the dialog driven system to a participant:

detecting, by a dialog driven system, voice audio;

generating, by the dialog driven system, an acoustic encoding of the voice audio using a pre-trained audio encoder machine learning model created with self-supervised fine-tuning, wherein the acoustic encoding comprises one or more acoustic features;

generating, by the dialog driven system, one or more context inclusive encodings for the voice audio comprising a text encoding of dialog not yet presented before the detected voice audio;

based, at least in part, on the acoustic encoding, including as inputs to an interruption classifier machine learning model the one or more acoustic features, and the one or more context inclusive encodings for the voice audio, including the text encoding of dialog not yet presented before the detected voice audio, applying, by the dialog driven system, the interruption classifier machine learning model that classifies the voice audio as a false interruption of the dialog; and

ignoring the voice audio and continuing, by the dialog driven system, the audio presentation of the dialog responsive to the false interruption classification of the voice audio.

6 . The method of claim 5 , further comprising combining the acoustic encoding with the one or more other context inclusive encodings, wherein the combination is provided as combined input to the machine learning model that classifies the voice audio.

7 . The method of claim 6 , wherein the combination is a weighted combination of the acoustic encoding and the one or more other context inclusive encodings.

8 . The method of claim 5 , wherein one of the one or more other context inclusion encodings is a text encoding of further dialog that was presented before the detection of the voice audio.

9 . The method of claim 5 , wherein one of the one or more other context inclusion encodings is an encoding indicating whether the dialog being presented is an elicitation to the participant.

10 . The method of claim 5 , wherein one of the one or more other context inclusion encodings is an encoding of prior participant interaction with the dialog driven system.

11 . The method of claim 5 , wherein the pre-trained audio encoder machine learning model created with self-supervised fine-tuning is a Hidden Unit Bidirectional Encoder Representations from Transforms (HuBERT) machine learning model.

12 . The method of claim 5 , further comprising:

detecting, by the dialog driven system, a second voice audio during an audio presentation of a second dialog generated by the dialog driven system to a second participant;

generating, by the dialog driven system, a second acoustic encoding of the second voice audio using the pre-trained audio encoder machine learning model created with self-supervised fine-tuning;

based, at least in part, on the second acoustic encoding, applying, by the dialog driven system, the machine learning model that classifies the second voice audio as noise; and

modifying, by the dialog driven system, the audio presentation of the second dialog responsive to the noise classification of the second voice audio.

13 . The method of claim 5 , further comprising:

detecting, by the dialog driven system, a second voice audio during an audio presentation of a second dialog generated by the dialog driven system to a second participant;

generating, by the dialog driven system, a second acoustic encoding of the second voice audio using the pre-trained audio encoder machine learning model created with self-supervised fine-tuning;

based, at least in part, on the second acoustic encoding, applying, by the dialog driven system, the machine learning model that classifies the second voice audio as a true interruption of the second dialog; and

stopping, by the dialog driven system, the audio presentation of the second dialog responsive to the true interruption classification of the second voice audio to accept the second voice audio as a response to the second dialog.

14 . One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement a dialog driven system that implements:

during an audio presentation of dialog generated by the dialog driven system to a participant:

detecting voice audio;

responsive to detecting the voice audio:

generating an acoustic encoding of the voice audio using a pre-trained audio encoder machine learning model created with self-supervised fine-tuning, wherein the acoustic encoding comprises one or more acoustic features;

generating one or more context inclusive encodings for the voice audio comprising a text encoding of dialog not yet presented before the detected voice audio;

based, at least in part, on the acoustic encoding, including as inputs to an interruption classifier machine learning model the one or more acoustic features, and the one or more context inclusive encodings, including the text encoding of dialog not yet presented before the detected voice audio, applying, by the dialog driven system, the interruption classifier machine learning model that classifies the voice audio as a false interruption of the dialog; and

ignoring and continuing the audio presentation of the dialog responsive to the false interruption classification of the voice audio.

15 . The one or more non-transitory, computer-readable storage media of claim 14 , storing further instructions that when executed on or across the one or more computing devices, cause the dialog driven system to further implement:

detecting a second voice audio during an audio presentation of a second dialog generated by the dialog driven system to a second participant;

generating a second acoustic encoding of the second voice audio using the pre-trained audio encoder machine learning model created with self-supervised fine-tuning;

based, at least in part, on the second acoustic encoding, applying the machine learning model that classifies the second voice audio as a true interruption of the second dialog; and

stopping the audio presentation of the second dialog responsive to the true interruption classification of the second voice audio to accept the second voice audio as a response to the second dialog.

16 . The one or more non-transitory, computer-readable storage media of claim 14 ,

wherein one of the context inclusive encodings is a text encoding of the dialog being presented using a text encoder machine learning model; and

wherein the one or more non-transitory computer-readable storage media store further instructions that when executed on or across the one or more computing devices, cause the dialog driven system to further implement:

combining the acoustic encoding with the text encoding, wherein the combination is provided as combined input to the machine learning model that classifies the voice audio.

17 . The one or more non-transitory, computer-readable storage media of claim 14 ,

wherein one of the context inclusive encodings is a dialog encoding indicating whether the dialog being presented is an elicitation to the participant; and

wherein the one or more non-transitory computer-readable storage media store further instructions that when executed on or across the one or more computing devices, cause the dialog driven system to further implement:

combining the acoustic encoding with the dialog encoding, wherein the combination is provided as combined input to the machine learning model that classifies the voice audio.

18 . The one or more non-transitory, computer-readable storage media of claim 14 ,

wherein one of the context inclusive encodings is a participant history encoding indicating prior participant interaction with the dialog driven system; and

wherein the one or more non-transitory computer-readable storage media store further instructions that when executed on or across the one or more computing devices, cause the dialog driven system to further implement:

combining the acoustic encoding with the participant history encoding, wherein the combination is provided as combined input to the machine learning model that classifies the voice audio.

19 . The one or more non-transitory, computer-readable storage media of claim 14 , wherein a weighted combination of the acoustic audio and the one or more other context inclusive encodings is provided as combined input to the machine learning model that classifies the voice audio.

20 . The one or more non-transitory, computer-readable storage media of claim 14 , wherein the dialog driven system is an application hosted as part of a dialog driven application management service and wherein the dialog is generated as part of an interaction sequence specified for the application via an interface of the dialog driven application management service.