Adversarial speech-text protection against automated analysis
View Patent ↗A method, system, and computer program product are disclosed. The method includes processing an audio signal that includes speech data and transcribing the speech data to generate text data. The method also includes identifying a vulnerable portion of the text data and, in response, applying adversarial text to the text data to generate robust text data. Adversarial noise corresponding to the robust text data is generated and applied to the speech data.
1. A method, comprising:
processing an audio signal comprising speech data;
transcribing the speech data to generate text data;
identifying a vulnerable portion of the text data;
in response to the identifying, modifying the text data to generate a robust transcript, wherein the modifying comprises replacing the vulnerable portion of the text data with adversarial text;
designing adversarial noise corresponding to the adversarial text; and
applying the corresponding adversarial noise to the audio signal to generate a robust audio signal comprising modified speech data that, when transcribed, generates a transcript with a similarity to the robust transcript that is above a threshold similarity.
2. The method of claim 1 , wherein the corresponding adversarial noise comprises adversarial speech data designed to cause speech-to-text models to produce a targeted transcription of the modified speech data matching the robust transcript.
3. The method of claim 2 , wherein the targeted transcription is at least 90% similar to the robust transcript.
4. The method of claim 1 , wherein the identifying the vulnerable portion comprises generating importance scores for portions of the text data, and wherein the vulnerable portion is a portion of the text data with a highest importance score.
5. The method of claim 1 , further comprising selecting the adversarial text, wherein the selecting the adversarial text comprises:
identifying, in an embedding space of a word from the vulnerable portion, a word that is semantically equivalent to the word from the vulnerable portion; and
determining that replacing the word from the vulnerable portion with the identified semantically equivalent word reduces a confidence score associated with a prediction of a target machine-learning model.
6. The method of claim 1 , wherein the audio signal further comprises adversarial non-speech noise.
7. The method of claim 6 , wherein the processing the audio signal comprises removing the adversarial non-speech noise prior to transcribing the speech data.
8. The method of claim 1 , wherein the replacing the vulnerable portion comprises using a word embedding process to replace a word in the vulnerable portion with a semantically equivalent word.
9. The method of claim 8 , wherein the applying the corresponding adversarial noise replaces an utterance of the word in the audio signal with an utterance of the semantically equivalent word.
10. The method of claim 1 , wherein:
the modifying the text data comprises deleting a word from the vulnerable portion; and
the applying the corresponding adversarial noise comprises deleting an utterance of the word from the audio signal.
11. A system, comprising:
a memory; and
a processor communicatively coupled to the memory, wherein the processor is configured to perform a method comprising:
processing an audio signal comprising speech data;
transcribing the speech data to generate text data;
identifying a vulnerable portion of the text data;
in response to the identifying, modifying the text data to generate a robust transcript, wherein the modifying comprises replacing the vulnerable portion of the text data with adversarial text;
designing adversarial noise corresponding to the adversarial text; and
applying the corresponding adversarial noise to the audio signal to generate a robust audio signal comprising modified speech data that, when transcribed, generates a transcript with a similarity to the robust transcript that is above a threshold similarity.
12. The system of claim 11 , wherein the corresponding adversarial noise comprises adversarial speech data designed to cause speech-to-text models to produce a targeted transcription of the modified speech data matching the robust transcript.
13. The system of claim 11 , wherein the identifying the vulnerable portion comprises generating importance scores for portions of the text data, and wherein the vulnerable portion is a portion of the text data with a highest importance score.
14. The system of claim 11 , further comprising selecting the adversarial text, wherein the selecting the adversarial text comprises:
identifying, in an embedding space of a word from the vulnerable portion, a word that is semantically equivalent to the word from the vulnerable portion; and
determining that replacing the word from the vulnerable portion with the identified semantically equivalent word reduces a confidence score associated with a prediction of a target machine-learning model.
15. The system of claim 11 , wherein the audio signal further comprises adversarial non-speech noise.
16. The system of claim 15 , wherein the processing the audio signal comprises removing the adversarial non-speech noise prior to transcribing the speech data.
17. A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause a device to perform a method, the method comprising:
processing an audio signal comprising speech data;
transcribing the speech data to generate text data;
identifying a vulnerable portion of the text data;
in response to the identifying, modifying the text data to generate a robust transcript, wherein the modifying comprises replacing the vulnerable portion of the text data with adversarial text;
designing adversarial noise corresponding to the adversarial text; and
applying the corresponding adversarial noise to the audio signal to generate a robust audio signal comprising modified speech data that, when transcribed, generates a transcript with a similarity to the robust transcript that is above a threshold similarity.
18. The computer program product of claim 17 , wherein the identifying the vulnerable portion comprises generating importance scores for portions of the text data, and wherein the vulnerable portion is a portion of the text data with a highest importance score.
19. The computer program product of claim 17 , further comprising selecting the adversarial text, wherein the selecting the adversarial text comprises:
identifying, in an embedding space of a word from the vulnerable portion, a word that is semantically equivalent to the word from the vulnerable portion; and
determining that replacing the word from the vulnerable portion with the identified semantically equivalent word reduces a confidence score associated with a prediction of a target machine-learning model.
20. The computer program product of claim 17 , wherein the processing the audio signal comprises removing adversarial non-speech noise prior to transcribing the speech data.