Granted Patent B2

US 11,990,134 · App. 18/220,632 · Granted May 21, 2024

Inventors: Xiaolong Li (Cary, NC); Xiaozhuo Cheng (Cary, NC); Xu Yang (Cary, NC)

Assignee: SAS INSTITUTE INC.

G10L15/26G10L15/02G10L15/04G10L25/30G10L25/78G10L2025/783

Abstract

Claims (144)

1. A computer-program product embodied in a non-transitory machine-readable storage medium storing computer instructions that, when executed by one or more processors, perform operations comprising:

constructing a transcript adaptation training data corpus comprising a plurality of transcript normalization training data samples, wherein each of the plurality of transcript normalization training data samples includes:

a training sample pairing between (i) a predicted audio transcript that includes at least one numerical expression and (ii) an adapted audio transcript that includes an alphabetic representation of the at least one numerical expression;

a transcript normalization identifier that, when applied to a model input comprising a target audio transcript, defines a text-to-text transformation objective causing a numeric-to-alphabetic expression machine learning model to predict an alphabetic-equivalent audio transcript that represents each numerical expression included in the target audio transcript in one or more alphabetic tokens;

configuring the numeric-to-alphabetic expression machine learning model based on a training of a machine learning text-to-text transformer model using the transcript adaptation training data corpus; and

executing the numeric-to-alphabetic expression machine learning model within a speech-to-text post-processing sequence of a speech-to-text service based on the numeric-to-alphabetic expression machine learning model satisfying a minimum audio transcript adaptation efficacy value;

obtaining audio data comprising one or more utterances;

generating, via a speech-to-text machine learning model, a probable audio transcript based on an input of the audio data, wherein the probable audio transcript includes a plurality of numerical expressions;

generating, via the numeric-to-alphabetic expression machine learning model, an adjusted audio transcript of the probable audio transcript based on an input of a task-specific instruction to the numeric-to-alphabetic expression machine learning model, wherein the task-specific instruction includes:

an instructional prefix component comprising the transcript normalization identifier, wherein the numeric-to-alphabetic expression machine learning model identifies a task type of the instructional prefix component, wherein the task type of the instructional prefix component corresponds to the transcript normalization identifier; and

an input text string comprising the probable audio transcript; and

obtaining, from a memory, a set of weights and biases generated from the training of the machine learning text-to-text transformer model that corresponds to the transcript normalization identifier, wherein the executing the numeric-to-alphabetic expression machine learning model includes using the set of weights and biases to generate the adjusted audio transcript.

2. The computer-program product according to claim 1 , wherein:

a subset of the plurality of transcript normalization training data samples includes a plurality of ambiguous transcript normalization training data samples, wherein each predicted audio transcript of the plurality of ambiguous transcript normalization training data samples includes at least one ambiguous numerical expression.

3. The computer-program product according to claim 1 , wherein:

the at least one numerical expression of a target predicted audio transcript includes an ambiguous numerical expression; and

the ambiguous numerical expression includes a numerical expression having a plurality of distinct, possible alphabetic representations.

4. The computer-program product according to claim 1 , further comprising:

receiving a target audio data file comprising one or more utterances associated with a target user;

generating, via a speech-to-text machine learning model, a likely audio transcript of the target audio data file based on an input of the target audio data file to the speech-to-text machine learning model, wherein the likely audio transcript includes a numerical expression; and

generating, via the numeric-to-alphabetic expression machine learning model, a transformed audio transcript that semantically represents the numerical expression of the likely audio transcript in an alphabetic-equivalent state based on an input of the likely audio transcript to the numeric-to-alphabetic expression machine learning model.

5. The computer-program product according to claim 1 , wherein:

each of the plurality of transcript normalization training data samples includes:

a first component comprising the transcript normalization identifier and the predicted audio transcript, wherein the transcript normalization identifier is a prefix to the predicted audio transcript; and

a second component comprising the adapted audio transcript.

6. The computer-program product according to claim 1 , wherein:

the machine learning text-to-text transformer model is configured to perform a plurality of trained text-to-text transformation tasks;

the transcript normalization identifier defines an audio transcript normalization task that is distinct from the plurality of trained text-to-text transformation tasks; and

the numeric-to-alphabetic expression machine learning model is configured to perform the plurality of trained text-to-text transformation tasks and the audio transcript normalization task based on the training of the machine learning text-to-text transformer model with the transcript adaptation training data corpus.

7. The computer-program product according to claim 1 , wherein:

the plurality of numerical expressions of the probable audio transcript includes a first numerical expression of a first numerical expression type and a second numerical expression of a second numerical expression type;

the numeric-to-alphabetic expression machine learning model converts:

the first numerical expression of the probable audio transcript to a first alphabetic-equivalent representation based on the first numerical expression type; and

the second numerical expression of the probable audio transcript to a second alphabetic-equivalent representation based on the second numerical expression type; and

the adjusted audio transcript includes the first alphabetic-equivalent representation and the second alphabetic-equivalent representation.

8. The computer-program product according to claim 1 , wherein:

the speech-to-text service includes a speech-to-text machine learning model and the numeric-to-alphabetic expression machine learning model; and

executing the numeric-to-alphabetic expression machine learning model within the speech-to-text post-processing sequence of the speech-to-text service includes:

providing the model input comprising the target audio transcript to the numeric-to-alphabetic expression machine learning model;

based on providing the model input to the numeric-to-alphabetic expression machine learning model:

identifying, via the numeric-to-alphabetic expression machine learning model, a likely numerical character in the target audio transcript;

generating, via the numeric-to-alphabetic expression machine learning model, the alphabetic-equivalent audio transcript in which one or more alphabetic characters semantically replaces the likely numerical character; and

outputting, via the numeric-to-alphabetic expression machine learning model, the alphabetic-equivalent audio transcript.

9. The computer-program product according to claim 1 , further comprising:

generating, via a speech-to-text machine learning model, a raw audio transcript based on an input of audio data to the speech-to-text machine learning model; and

commencing the speech-to-text post-processing sequence in response to the generating of the raw audio transcript, wherein the speech-to-text post-processing sequence includes:

obtaining the raw audio transcript;

configuring a task-specific instruction that includes the transcript normalization identifier and the raw audio transcript in response to obtaining the raw audio transcript;

providing the task-specific instruction as model input to the numeric-to-alphabetic expression machine learning model; and

generating a normalized audio transcript based on the model input of the task-specific instruction to the numeric-to-alphabetic expression machine learning model.

10. The computer-program product according to claim 1 , wherein:

the numeric-to-alphabetic expression machine learning model satisfies the minimum audio transcript adaptation efficacy value when a predictive accuracy value of the numeric-to-alphabetic expression machine learning model satisfies a predetermined, minimum audio transcript normalization accuracy value.

11. A computer-program product embodied in a non-transitory machine-readable storage medium storing computer instructions that, when executed by one or more processors, perform operations comprising:

constructing a transcript adaptation training data corpus comprising a plurality of transcript normalization training data samples, wherein each of the plurality of transcript normalization training data samples includes:

a training sample pairing between (i) a predicted audio transcript that includes at least one alphabetic expression and (ii) an adapted audio transcript that includes a numerical representation of the at least one alphabetic expression;

a transcript conversion identifier that, when applied to a model input comprising a target audio transcript, defines a text-to-text transformation objective causing an alphabetic-to-numerical expression machine learning model to predict a numerical-equivalent audio transcript that represents a target alphabetic expression included in the target audio transcript in one or more numeric tokens;

configuring the alphabetic-to-numerical expression machine learning model based on a training of a machine learning text-to-text transformer model using the transcript adaptation training data corpus; and

executing the alphabetic-to-numerical expression machine learning model within a speech-to-text post-processing sequence of a speech-to-text service based on the alphabetic-to-numerical expression machine learning model satisfying a minimum audio transcript adaptation efficacy value;

obtaining audio data comprising one or more utterances;

generating, via a speech-to-text machine learning model, a probable audio transcript based on an input of the audio data, wherein the probable audio transcript includes a plurality of numerical expressions;

generating, via the alphabetic-to-numerical expression machine learning model, an adjusted audio transcript of the probable audio transcript based on an input of a task-specific instruction to the alphabetic-to-numerical expression machine learning model, wherein the task-specific instruction includes:

an instructional prefix component comprising the transcript conversion identifier, wherein the alphabetic-to-numerical expression machine learning model identifies a task type of the instructional prefix component, wherein the task type of the instructional prefix component corresponds to the transcript conversion identifier; and

an input text string comprising the probable audio transcript; and

obtaining, from a memory, a set of weights and biases generated from the training of the machine learning text-to-text transformer model that corresponds to the transcript conversion identifier, wherein the executing the alphabetic-to-numerical expression machine learning model includes using the set of weights and biases to generate the adjusted audio transcript.

12. A computer-implemented method comprising:

constructing a transcript adaptation training data corpus comprising a plurality of transcript normalization training data samples, wherein each of the plurality of transcript normalization training data samples includes:

a training sample pairing between (i) a predicted audio transcript that includes at least one numerical expression and (ii) an adapted audio transcript that includes an alphabetic representation of the at least one numerical expression;

a transcript normalization identifier that, when applied to a model input comprising a target audio transcript, defines a text-to-text transformation objective causing a numeric-to-alphabetic expression machine learning model to predict an alphabetic-equivalent audio transcript that represents each numerical expression included in the target audio transcript in one or more alphabetic tokens;

configuring the numeric-to-alphabetic expression machine learning model based on a training of a machine learning text-to-text transformer model using the transcript adaptation training data corpus; and

executing the numeric-to-alphabetic expression machine learning model within a speech-to-text post-processing sequence of a speech-to-text service based on the numeric-to-alphabetic expression machine learning model satisfying a minimum audio transcript adaptation efficacy value;

obtaining audio data comprising one or more utterances;

generating, via a speech-to-text machine learning model, a probable audio transcript based on an input of the audio data, wherein the probable audio transcript includes a plurality of numerical expressions;

generating, via the numeric-to-alphabetic expression machine learning model, an adjusted audio transcript of the probable audio transcript based on an input of a task-specific instruction to the numeric-to-alphabetic expression machine learning model, wherein the task-specific instruction includes:

an instructional prefix component comprising the transcript normalization identifier, wherein the numeric-to-alphabetic expression machine learning model identifies a task type of the instructional prefix component, wherein the task type of the instructional prefix component corresponds to the transcript normalization identifier; and

an input text string comprising the probable audio transcript; and

obtaining, from a memory, a set of weights and biases generated from the training of the machine learning text-to-text transformer model that corresponds to the transcript normalization identifier, wherein the executing the numeric-to-alphabetic expression machine learning model includes using the set of weights and biases to generate the adjusted audio transcript.

13. The computer-implemented method according to claim 12 , wherein:

a subset of the plurality of transcript normalization training data samples includes a plurality of ambiguous transcript normalization training data samples, wherein each predicted audio transcript of the plurality of ambiguous transcript normalization training data samples includes at least one ambiguous numerical expression.

14. The computer-implemented method according to claim 12 , wherein:

the at least one numerical expression of a target predicted audio transcript includes an ambiguous numerical expression; and

the ambiguous numerical expression includes a numerical expression having a plurality of distinct, possible alphabetic representations.

15. The computer-implemented method according to claim 12 , further comprising:

receiving a target audio data file comprising one or more utterances associated with a target user;

generating, via a speech-to-text machine learning model, a likely audio transcript of the target audio data file based on an input of the target audio data file to the speech-to-text machine learning model, wherein the likely audio transcript includes a numerical expression; and

generating, via the numeric-to-alphabetic expression machine learning model, a transformed audio transcript that semantically represents the numerical expression of the likely audio transcript in an alphabetic-equivalent state based on an input of the likely audio transcript to the numeric-to-alphabetic expression machine learning model.

16. The computer-implemented method according to claim 14 , wherein:

each of the plurality of transcript normalization training data samples includes:

a first component comprising the transcript normalization identifier and the predicted audio transcript, wherein the transcript normalization identifier is a prefix to the predicted audio transcript; and

a second component comprising the adapted audio transcript.

17. The computer-implemented method according to claim 12 , wherein:

the machine learning text-to-text transformer model is configured to perform a plurality of trained text-to-text transformation tasks;

the transcript normalization identifier defines an audio transcript normalization task that is distinct from the plurality of trained text-to-text transformation tasks; and

the numeric-to-alphabetic expression machine learning model is configured to perform the plurality of trained text-to-text transformation tasks and the audio transcript normalization task based on the training of the machine learning text-to-text transformer model with the transcript adaptation training data corpus.

18. The computer-implemented method according to claim 12 , wherein:

the plurality of numerical expressions of the probable audio transcript includes a first numerical expression of a first numerical expression type and a second numerical expression of a second numerical expression type;

the numeric-to-alphabetic expression machine learning model converts:

the first numerical expression of the probable audio transcript to a first alphabetic-equivalent representation based on the first numerical expression type; and

the second numerical expression of the probable audio transcript to a second alphabetic-equivalent representation based on the second numerical expression type; and

the adjusted audio transcript includes the first alphabetic-equivalent representation and the second alphabetic-equivalent representation.

19. The computer-implemented method according to claim 12 , wherein:

the speech-to-text service includes a speech-to-text machine learning model and the numeric-to-alphabetic expression machine learning model; and

executing the numeric-to-alphabetic expression machine learning model within the speech-to-text post-processing sequence of the speech-to-text service includes:

providing the model input comprising the target audio transcript to the numeric-to-alphabetic expression machine learning model;

based on providing the model input to the numeric-to-alphabetic expression machine learning model:

identifying, via the numeric-to-alphabetic expression machine learning model, a likely numerical character in the target audio transcript;

generating, via the numeric-to-alphabetic expression machine learning model, the alphabetic-equivalent audio transcript in which one or more alphabetic characters semantically replaces the likely numerical character; and

outputting, via the numeric-to-alphabetic expression machine learning model, the alphabetic-equivalent audio transcript.

20. The computer-implemented method according to claim 12 , further comprising:

generating, via a speech-to-text machine learning model, a raw audio transcript based on an input of audio data to the speech-to-text machine learning model; and

commencing the speech-to-text post-processing sequence in response to the generating of the raw audio transcript, wherein the speech-to-text post-processing sequence includes:

obtaining the raw audio transcript;

configuring a task-specific instruction that includes the transcript normalization identifier and the raw audio transcript in response to obtaining the raw audio transcript;

providing the task-specific instruction as model input to the numeric-to-alphabetic expression machine learning model; and

generating a normalized audio transcript based on the model input of the task-specific instruction to the numeric-to-alphabetic expression machine learning model.

21. The computer-implemented method according to claim 12 , wherein:

the numeric-to-alphabetic expression machine learning model satisfies the minimum audio transcript adaptation efficacy value when a predictive accuracy value of the numeric-to-alphabetic expression machine learning model satisfies a predetermined, minimum audio transcript normalization accuracy value.

22. A computer-implemented system comprising:

one or more processors;

a memory;

a computer-readable medium operably coupled to the one or more processors, the computer-readable medium having computer-readable instructions stored thereon that, when executed by the one or more processors, cause a computing device to perform operations comprising:

constructing a transcript adaptation training data corpus comprising a plurality of transcript normalization training data samples, wherein each of the plurality of transcript normalization training data samples includes:

a training sample pairing between (i) a predicted audio transcript that includes at least one numerical expression and (ii) an adapted audio transcript that includes an alphabetic representation of the at least one numerical expression;

a transcript normalization identifier that, when applied to a model input comprising a target audio transcript, defines a text-to-text transformation objective causing a numeric-to-alphabetic expression machine learning model to predict an alphabetic-equivalent audio transcript that represents each numerical expression included in the target audio transcript in one or more alphabetic tokens;

configuring the numeric-to-alphabetic expression machine learning model based on a training of a machine learning text-to-text transformer model using the transcript adaptation training data corpus; and

executing the numeric-to-alphabetic expression machine learning model within a speech-to-text post-processing sequence of a speech-to-text service based on the numeric-to-alphabetic expression machine learning model satisfying a minimum audio transcript adaptation efficacy value;

obtaining audio data comprising one or more utterances;

generating, via a speech-to-text machine learning model, a probable audio transcript based on an input of the audio data, wherein the probable audio transcript includes a plurality of numerical expressions;

generating, via the numeric-to-alphabetic expression machine learning model, an adjusted audio transcript of the probable audio transcript based on an input of a task-specific instruction to the numeric-to-alphabetic expression machine learning model, wherein the task-specific instruction includes:

an instructional prefix component comprising the transcript normalization identifier, wherein the numeric-to-alphabetic expression machine learning model identifies a task type of the instructional prefix component, wherein the task type of the instructional prefix component corresponds to the transcript normalization identifier; and

an input text string comprising the probable audio transcript; and

obtaining, from a memory, a set of weights and biases generated from the training of the machine learning text-to-text transformer model that corresponds to the transcript normalization identifier, wherein the executing the numeric-to-alphabetic expression machine learning model includes using the set of weights and biases to generate the adjusted audio transcript.

23. The computer-implemented system according to claim 22 , wherein:

a subset of the plurality of transcript normalization training data samples includes a plurality of ambiguous transcript normalization training data samples, wherein each predicted audio transcript of the plurality of ambiguous transcript normalization training data samples includes at least one ambiguous numerical expression.

24. The computer-implemented system according to claim 22 , wherein:

the at least one numerical expression of a target predicted audio transcript includes an ambiguous numerical expression; and

the ambiguous numerical expression includes a numerical expression having a plurality of distinct, possible alphabetic representations.

25. The computer-implemented system according to claim 22 , further comprising:

receiving a target audio data file comprising one or more utterances associated with a target user;

generating, via a speech-to-text machine learning model, a likely audio transcript of the target audio data file based on an input of the target audio data file to the speech-to-text machine learning model, wherein the likely audio transcript includes a numerical expression; and

generating, via the numeric-to-alphabetic expression machine learning model, a transformed audio transcript that semantically represents the numerical expression of the likely audio transcript in an alphabetic-equivalent state based on an input of the likely audio transcript to the numeric-to-alphabetic expression machine learning model.

26. The computer-implemented system according to claim 22 , wherein:

each of the plurality of transcript normalization training data samples includes:

a first component comprising the transcript normalization identifier and the predicted audio transcript, wherein the transcript normalization identifier is a prefix to the predicted audio transcript; and

a second component comprising the adapted audio transcript.

Assignments (1)

Continuity (14)