Granted Patent B2

US 12,444,414 · App. 17/247,414 · Granted Oct 14, 2025

Inventors: Shikhar Kwatra (San Jose, CA); Komminist Weldemariam (Ottawa, CA); Zachary A. Silverstein (Austin, TX); Victor Povar (Vancouver, CA)

Assignee: INTERNATIONAL BUSINESS MACHINES CORPORATION

G10L15/22G06F9/453G06N3/08G10L25/63G10L2015/225

Abstract

Claims (37)

1. A method for dynamic speech modulation, the method comprising:

extracting, by a digital voice assistant, Mel-frequency cepstral coefficient (MFCC) features from a received audio command from a user to identify the user and a corresponding user profile based on the MFCC features, wherein the corresponding user profile stores a default language of the user and one or more user pronunciation models learned by the digital voice assistant based on analysis of user pronunciations;

generating, by the digital voice assistant, a first response to the received audio command, wherein the first response includes a first pronunciation associated with a different language than the default language of the user;

sampling, by the digital voice assistant, the first response to determine whether the first response will be understood by the user based on a comprehension level of the user associated with the different language in the first response, wherein the comprehension level of the user is identified in the corresponding user profile, and wherein the comprehension level is dynamically set by the digital voice assistant based on learning a user comprehension rate for one or more prior unmodified responses from the digital voice assistant; and

in response to determining, by the digital voice assistant, that the first response includes a lower confidence score than the comprehension level of the user, performing a cosine similarity to identify at least one speech feature in the first pronunciation of the first response that is different relative to the one or more user pronunciation models learned by the digital voice assistant, modifying the at least one speech feature in the first pronunciation of the first response to align the first pronunciation with the one or more user pronunciation models learned by the digital voice assistant, and transmitting to the user, by the digital voice assistant, a second response to the received audio command generated based on the one or more user pronunciation models, wherein the second response includes a higher confidence score than the comprehension level of the user.

2. The method of claim 1 , further comprising:

transmitting to the user, by the digital voice assistant, the first response to the received audio command to determine whether the first response will be understood by the user; and

determining that the first response transmitted to the user is not understood by the user based on determining, by the digital voice assistant executing a sentiment analysis application programming interface (API) and a tone analysis API on a user response to the first response, that a frustration level of the user exceeds a baseline frustration level.

3. The method of claim 1 , further comprising:

using a long short-term memory (LSTM) recurrent neural network (RNN), trained using historical user data including pronunciation data, accent data, and the corresponding user profile, to predict a comprehension difficulty of the user.

4. The method of claim 1 , further comprising:

updating the one or more user pronunciation models based on a perceived understanding of the second response by the user.

5. A computer system for dynamic speech modulation, comprising:

one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage medium, and program instructions stored on at least one of the one or more computer-readable tangible storage medium for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, wherein the computer system is capable of performing a method comprising:

extracting, by a digital voice assistant, Mel-frequency cepstral coefficient (MFCC) features from a received audio command from a user to identify the user and a corresponding user profile based on the MFCC features, wherein the corresponding user profile stores a default language of the user and one or more user pronunciation models learned by the digital voice assistant based on analysis of user pronunciations;

generating, by the digital voice assistant, a first response to the received audio command, wherein the first response includes a first pronunciation associated with a different language than the default language of the user;

sampling, by the digital voice assistant, the first response to determine whether the first response will be understood by the user based on a comprehension level of the user associated with the different language in the first response, wherein the comprehension level of the user is identified in the corresponding user profile, and wherein the comprehension level is dynamically set by the digital voice assistant based on learning a user comprehension rate for one or more prior unmodified responses from the digital voice assistant; and

in response to determining, by the digital voice assistant, that the first response includes a lower confidence score than the comprehension level of the user, performing a cosine similarity to identify at least one speech feature in the first pronunciation of the first response that is different relative to the one or more user pronunciation models learned by the digital voice assistant, modifying the at least one speech feature in the first pronunciation of the first response to align the first pronunciation with the one or more user pronunciation models learned by the digital voice assistant, and transmitting to the user, by the digital voice assistant, a second response to the received audio command generated based on the one or more user pronunciation models, wherein the second response includes a higher confidence score than the comprehension level of the user.

6. The computer system of claim 5 , further comprising:

transmitting to the user, by the digital voice assistant, the first response to the received audio command to determine whether the first response will be understood by the user; and

determining that the first response transmitted to the user is not understood by the user based on determining, by the digital voice assistant executing a sentiment analysis application programming interface (API) and a tone analysis API on a user response to the first response, that a frustration level of the user exceeds a baseline frustration level.

7. The computer system of claim 5 , further comprising:

using a long short-term memory (LSTM) recurrent neural network (RNN), trained using historical user data including pronunciation data, accent data, and the corresponding user profile, to predict a comprehension difficulty of the user.

8. The computer system of claim 5 , further comprising:

updating the one or more user pronunciation models based on a perceived understanding of the second response by the user.

9. A computer program product for dynamic speech modulation, comprising:

one or more non-transitory computer-readable storage media and program instructions stored on at least one of the one or more non-transitory computer-readable storage media, the program instructions executable by a processor to cause the processor to perform a method comprising:

extracting, by a digital voice assistant, Mel-frequency cepstral coefficient (MFCC) features from a received audio command from a user to identify the user and a corresponding user profile based on the MFCC features, wherein the corresponding user profile stores a default language of the user and one or more user pronunciation models learned by the digital voice assistant based on analysis of user pronunciations;

generating, by the digital voice assistant, a first response to the received audio command, wherein the first response includes a first pronunciation associated with a different language than the default language of the user;

sampling, by the digital voice assistant, the first response to determine whether the first response will be understood by the user based on a comprehension level of the user associated with the different language in the first response, wherein the comprehension level of the user is identified in the corresponding user profile, and wherein the comprehension level is dynamically set by the digital voice assistant based on learning a user comprehension rate for one or more prior unmodified responses from the digital voice assistant; and

in response to determining, by the digital voice assistant, that the first response includes a lower confidence score than the comprehension level of the user, performing a cosine similarity to identify at least one speech feature in the first pronunciation of the first response that is different relative to the one or more user pronunciation models learned by the digital voice assistant, modifying the at least one speech feature in the first pronunciation of the first response to align the first pronunciation with the one or more user pronunciation models learned by the digital voice assistant, and transmitting to the user, by the digital voice assistant, a second response to the received audio command generated based on the one or more user pronunciation models, wherein the second response includes a higher confidence score than the comprehension level of the user.

10. The computer program product of claim 9 , further comprising:

transmitting to the user, by the digital voice assistant, the first response to the received audio command to determine whether the first response will be understood by the user; and

determining that the first response transmitted to the user is not understood by the user based on determining, by the digital voice assistant executing a sentiment analysis application programming interface (API) and a tone analysis API on a user response to the first response, that a frustration level of the user exceeds a baseline frustration level.

11. The computer program product of claim 9 , further comprising:

using a long short-term memory (LSTM) recurrent neural network (RNN), trained using historical user data including pronunciation data, accent data, and the corresponding user profile, to predict a comprehension difficulty of the user.

12. The computer program product of claim 9 , wherein the one or more user pronunciation models is updated based on a perceived understanding of the second response by the user.

Assignments (1)

Continuity (1)

References Cited (39)