Granted Patent B2

US 12,367,859 · App. 17/809,202 · Granted Jul 22, 2025

Inventors: Shreya Khare (Bangalore, IN); Ashish R. Mittal (Bengaluru, IN); Saneem Ahmed Chemmengath (Bangalore, IN); Samarth Bharadwaj (Bangalore, IN); Karthik Sankaranarayanan (Bangalore, IN)

Assignee: International Business Machines Corporation

G10L15/01G10L15/16G10L15/30

Abstract

Claims (47)

1. A computer-implemented method, comprising:

receiving audio data for a user, the audio data containing human speech;

generating, using a first speech to text model, text data representing the human speech of the audio data;

identifying a set of transcription errors of the first speech to text model;

generating a set of artificial intelligence (AI) factsheets describing model metadata for the first speech to text model, wherein at least one of the set of AI factsheets is a confidence sheet including a local explanation of the first speech to text model that provides values of audio quality, transcript quality, stylization and data redaction metrics, wherein the set of AI factsheets is a plurality of AI factsheets and generating the set of AI factsheets further comprises:

generating a first AI factsheet for the first speech to text model, the first AI factsheet being a model factsheet; and

generating a second AI factsheet for the first speech to text model, the second AI factsheet being a transcript factsheet; and

based on the first speech to text model, the set of transcription errors, the text data, the first AI factsheet, and the second AI factsheet, generating a second speech to text model customized to the user, wherein the first speech to text model identifies training data or subsets thereof for input to train the second speech to text model, and selecting the training data to address deficiencies represented by the set of transcription errors identified from the first speech to text model.

2. The method of claim 1 , wherein the set of transcription errors are a set of automatic speech recognition (ASR) errors and wherein identifying the set of ASR errors further comprises:

determining a set of characteristics for the set of ASR errors; and

attributing the set of ASR errors to a set of speech features of the audio data based on the set of characteristics.

3. The method of claim 2 , wherein identifying the set of ASR errors further comprises:

clustering ASR errors of the set of ASR errors to generate ASR error clusters, each ASR error cluster associated with at least one speech feature of the audio data.

4. The method of claim 1 , further comprising:

determining a customization level for the second speech to text model based on the set of AI factsheets.

5. A system, comprising:

one or more processors; and

a computer-readable storage medium, coupled to the one or more processors, storing program instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving audio data for a user, the audio data containing human speech;

generating, using a first speech to text model, text data representing the human speech of the audio data;

identifying a set of transcription errors of the first speech to text model;

generating a set of artificial intelligence (AI) factsheets describing model metadata for the first speech to text model, wherein at least one of the set of AI factsheets is a confidence sheet including a local explanation of the first speech to text model that provides values of audio quality, transcript quality, stylization and data redaction metrics, wherein the set of AI factsheets is a plurality of AI factsheets and generating the set of AI factsheets further comprises:

generating a first AI factsheet for the first speech to text model, the first AI factsheet being a model factsheet; and

generating a second AI factsheet for the first speech to text model, the second AI factsheet being a transcript factsheet; and

based on the first speech to text model, the set of transcription errors, the text data, the first AI factsheet, and the second AI factsheet, generating a second speech to text model customized to the user, wherein the first speech to text model identifies training data or subsets thereof for input to train the second speech to text model, and selecting the training data to address deficiencies represented by the set of transcription errors identified from the first speech to text model.

6. The system of claim 5 , wherein the set of transcription errors are a set of automatic speech recognition (ASR) errors and wherein identifying the set of ASR errors further comprises:

determining a set of characteristics for the set of ASR errors; and

attributing the set of ASR errors to a set of speech features of the audio data based on the set of characteristics.

7. The system of claim 6 , wherein identifying the set of ASR errors further comprises:

clustering ASR errors of the set of ASR errors to generate ASR error clusters, each ASR error cluster associated with at least one speech feature of the audio data.

8. The system of claim 5 , wherein the operations further comprise:

determining a customization level for the second speech to text model based on the set of AI factsheets.

9. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions being executable by one or more processors to cause the one or more processors to perform operations comprising:

receiving audio data for a user, the audio data containing human speech;

generating, using a first speech to text model, text data representing the human speech of the audio data;

identifying a set of transcription errors of the first speech to text model;

generating a set of artificial intelligence (AI) factsheets describing model metadata for the first speech to text model, wherein at least one of the set of AI factsheets is a confidence sheet including a local explanation of the first speech to text model that provides values of audio quality, transcript quality, stylization and data redaction metrics, wherein the set of AI factsheets is a plurality of AI factsheets and generating the set of AI factsheets further comprises:

generating a first AI factsheet for the first speech to text model, the first AI factsheet being a model factsheet; and

generating a second AI factsheet for the first speech to text model, the second AI factsheet being a transcript factsheet; and

based on the first speech to text model, the set of transcription errors, the text data, the first AI factsheet, and the second AI factsheet, generating a second speech to text model customized to the user, wherein the first speech to text model identifies training data or subsets thereof for input to train the second speech to text model, and selecting the training data to address deficiencies represented by the set of transcription errors identified from the first speech to text model.

10. The computer program product of claim 9 , wherein the set of transcription errors are a set of automatic speech recognition (ASR) errors and wherein identifying the set of ASR errors further comprises:

determining a set of characteristics for the set of ASR errors; and

attributing the set of ASR errors to a set of speech features of the audio data based on the set of characteristics.

11. The computer program product of claim 10 , wherein identifying the set of ASR errors further comprises:

clustering ASR errors of the set of ASR errors to generate ASR error clusters, each ASR error cluster associated with at least one speech feature of the audio data.

12. The computer program product of claim 9 , wherein the operations further comprise:

determining a customization level for the second speech to text model based on the set of AI factsheets.

Assignments (1)

Continuity (1)

References Cited (23)