Detection of calls from voice assistants
Embodiments described herein provide for automatically classifying the types of devices that place calls to a call center. A call center system can detect whether an incoming call originated from voice assistant device using trained classification models received from a call analysis service. Embodiments described herein provide for methods and systems in which a computer executes machine learning algorithms that programmatically train (or otherwise generate) global or tailored classification models based on the various types of features of an audio signal and call data. A classification model is deployed to one or more call centers, where the model is used by call center computers executing classification processes for determining whether incoming telephone calls originated from a voice assistant device, such as Amazon Alexa® and Google Home®, or another type of device (e.g., cellular/mobile phone, landline phone, VoIP).
1. A computer-implemented method for detecting certain types of caller devices, the method comprising:
receiving, by a computer, call data for an incoming call, the call data comprising an audio signal and metadata;
extracting, by a computer, one or more feature-types from the audio signal of the call data for the incoming call, the one or more feature-types including a reverberation-based feature;
generating, by the computer using a classifier model applied on the one or more feature-types including the reverberation-based feature extracted from the call data of the incoming call, a device-type score for the incoming call based upon the one or more feature-types and the metadata of the call data for the incoming call, wherein the classifier model is trained to detect a type of device that originated a corresponding call according to the call data for a plurality of calls, the call data including the reverberation-based feature; and
determining, by the computer, that the incoming call originated from a voice assistant device in response to determining that the device-type score for the incoming call satisfies a threshold value.
2. The method according to claim 1 , wherein extracting the one or more feature-types comprises, for at least one feature-type:
generating, by the computer, a plurality of audio segments parsed from the audio signal of the incoming call; and
extracting, by the computer, one or more features of the at least one feature-type from the plurality of audio segments.
3. The method according to claim 2 , wherein a feature-type of the at least one feature-type is selected from the group consisting of a reverberation-based feature or features, one or more acoustic features, and sound-to-noise ratio features.
4. The method according to claim 2 , wherein the computer generates the plurality of segments by executing a voice activity detector program configured to generate the plurality of segments parsed from the audio signal, wherein each segment of the plurality of segments generated by the voice activity detector program contains a spoken utterance.
5. The method according to claim 1 , wherein extracting the one or more feature-types comprises extracting, by the computer, one or more features of the at least one feature-type based upon an entire audio signal.
6. The method according to claim 5 , wherein a feature-type of the at least one feature-type is selected from the group consisting of a spectral feature, custom mel frequency cepstral coefficients (MFCC) spectral features, a harmonic mean spectral feature, and a short-window correlation acoustic feature.
7. The method according to claim 1 , wherein the classifier model is associated with one or more standardization parameters, and wherein generating the device-type score for the incoming call using the classifier model comprises standardizing, by the computer, values of each respective feature-type according to a standardization parameter corresponding respectively to each feature-type.
8. The method according to claim 1 , wherein receiving the call data for the incoming call comprises storing, into a database, the call data for each respective call of the plurality of calls.
9. The method according to claim 1 , wherein the call data for the plurality of calls used to train the classification model includes at least one call originated from a voice assistant device.
10. The method according to claim 1 , further comprising generating, by the computer, an indicator via a GUI of a client computer that the incoming call originated from the voice assistant device in response to the computer determining that the device-type score satisfies the threshold.
11. The method according to claim 1 , wherein extracting one or more features of a feature-type from each respective call comprises:
generating, by the computer, one or more statistical parameters based upon a linear predictive coding residual of the audio signal; and
appending, by the computer, the one or more statistical parameters as a reverberation feature of the one or more features of the respective call data.
12. The method according to claim 1 , wherein extracting one or more features of a feature-type from each respective call comprises:
generating, by the computer, one or more statistical parameters based upon at least one of: a spectral rolloff of the audio signal, a spectral contrast of the audio signal, a spectral flatness of the audio signal, a spectral bandwidth of the audio signal, a spectral centroid of the audio signal, and a Fast Fourier Transform of the audio signal; and
appending, by the computer, the one or more statistical parameters as one or more spectral-based features of the one or more features of the respective call data.
13. A computer-implemented method for detecting certain types of caller devices, the method comprising:
receiving, by a computer, call data for a plurality of calls, the call data for each respective call comprising an audio signal and metadata of the respective call, wherein the plurality of calls includes one or more voice assistant calls that involved one or more voice assistant devices, the call data for each voice assistant call indicates a device-type is a voice assistant device;
for each respective call in the plurality of calls, extracting, by the computer, one or more feature-types from the audio signal of the call data including a reverberation-based feature;
training, by the computer, a classification model to detect a type of device that originated the corresponding call based on the one or more feature-types including the reverberation-based feature extracted from the call data of the plurality of calls, wherein the classifier model is trained to generate a device-type score according to one or more machine-learning algorithms used for the one or more feature-types;
generating, by the computer, one or more standardization parameters for the classification model, wherein each feature-type is normalized according to a corresponding standardization parameter;
storing, by the computer, the classification model in a machine-readable storage; and
identifying, by the computer, that an incoming call is a voice assistant call involving a voice assistant device based upon the device-type score for the incoming call generated by applying the classification model on the call data of an incoming call.
14. The method according to claim 13 , wherein extracting the one or more feature-types comprises, for the call data of each respective call:
generating, by the computer, a plurality of audio segments parsed from the audio signal of the respective call data; and
extracting, by the computer, one or more features of a feature-type from the plurality of audio segments.
15. The method according to claim 13 , wherein the feature-type is selected from the group consisting of a reverberation-based feature, one or more cepstral acoustic features, and a sound-to-noise ratio.
16. The method according to claim 13 , wherein the computer generates the plurality of segments from each respective call audio by executing a voice activity detector program configured to generate the plurality of segments parsed from the audio signal of the respective call data, wherein each segment of the plurality of segments generated by the voice activity detector program contains a spoken utterance.
17. The method according to claim 13 , wherein extracting the one or more feature-types comprises extracting, by the computer, one or more features of a feature-type based upon an entire audio signal.
18. The method according to claim 17 , wherein a feature-type of the at least one feature-type is selected from the group consisting of a spectral feature, custom mel frequency cepstral coefficients (MFCC) spectral features, a harmonic mean spectral feature, and a short-window correlation acoustic feature.
19. The method according to claim 13 , wherein receiving the call data for the plurality of calls comprises storing, into the database, the call data for each respective call of the plurality of calls.
20. The method according to claim 13 , wherein extracting one or more features of a feature-type from each respective call comprises:
generating, by the computer, one or more statistical parameters based upon a linear predictive coding residual of the audio signal; and
appending, by the computer, the one or more statistical parameters as a reverberation feature of the one or more features of the respective call data.
21. The method according to claim 13 , wherein extracting one or more features of a feature-type from each respective call comprises:
generating, by the computer, one or more statistical parameters based upon at least one of: a spectral rolloff of the audio signal, a spectral contrast of the audio signal, a spectral flatness of the audio signal, a spectral bandwidth of the audio signal, a spectral centroid of the audio signal, and a Fast Fourier Transform of the audio signal; and
appending, by the computer, the one or more statistical parameters as one or more spectral-based features of the one or more features of the respective call data.
22. The method according to claim 13 , wherein extracting one or more features of a feature type from each respective call comprises:
extracting, by the computer, mel frequency cepstral coefficients of the audio signal; and
appending, by the computer, the mel frequency cepstral coefficients as one or more spectral-based features of the respective call data.
23. The method according to claim 13 , wherein extracting one or more features of a feature type from each respective call comprises:
generating, by the computer one or more metrics of a signal-to-noise ratio of the audio signal; and
appending, by the computer, the one or more metrics as a noise-based feature of the one or more features of the respective call data.
24. The method according to claim 13 , wherein extracting one or more features of a feature type from each respective call comprises:
generating, by the computer, a short-window correlation measurement from the audio signal; and
appending, by the computer, the short-window correlation measurement as an acoustic feature of the one or more features of the respective call data.
25. The method according to claim 13 , for each respective voice assistant call in the plurality of calls:
receiving, by the computer from a client computer, an indicator that the voice assistant call originated from a voice assistant device.