Granted Patent B2

US 12,417,381 · App. 17/118,139 · Granted Sep 16, 2025

Inventors: Toru Nagano (Taito-ku, JP); Takashi Fukuda (Tokyo, JP); Gakuto Kurata (Tokyo, JP)

Assignee: International Business Machines Corporation

G06N3/08G06F18/214G06N5/02G10L15/02G10L15/16G10L2015/025

Abstract

Claims (66)

1. A computer-implemented method for training a student model via knowledge distillation with supplemented labeled training data, the computer-implemented method comprising:

receiving, by one or more processors, an acoustic feature with one or more associated reference labels;

generating, by the one or more processors, a first soft label for the acoustic feature based on a teacher model, wherein

the first soft label is a probability prediction for the acoustic feature which places a word or a word segment in a specific class for which the teacher model is designed to classify;

identifying, by the one or more processors, one or more candidate soft labels based on each reference label of the one or more associated reference labels, wherein the identified one or more candidate soft labels are from a soft label pool;

determining, by the one or more processors, if each of the identified one or more candidate soft labels is similar to the acoustic feature, wherein the identified one or more candidate soft labels is similar to the acoustic feature if the identified one or more candidate soft labels has the same sequence of phonemes as the acoustic feature;

generating, by the one or more processors, a similarity score for each of the identified one or more candidate soft labels, wherein the similarity score is generated based on the determination that each of the identified one or more candidate soft labels is similar to the acoustic feature;

determining, by the one or more processors, a second soft label, wherein

the identified one or more candidate soft labels include the second soft label, and

the similarity score of the second soft label is above a specific threshold; and

responsive to the determining of the second soft label:

outputting, by the one or more processors, the second soft label;

training, by the one or more processors, the student model using the generated first soft label and the second soft label, wherein the student model is an automatic speech recognition system; and

operating, by the one or more processors, the trained student model as a speech recognition system on a user device.

2. The computer-implemented method of claim 1 , further comprising generating the soft label pool by the teacher model.

3. The computer-implemented method of claim 1 , further comprising generating, by an acoustic feature generator, the acoustic feature, wherein the acoustic feature generator is a convolutional neural network.

4. The computer-implemented method of claim 1 , wherein the teacher model is a visual geometry group model.

5. The computer-implemented method of claim 1 , further comprising:

generating, by the one or more processors, a hard label for the acoustic feature; and

training, by the one or more processors, the student model with the hard label.

6. A system for generating labeled training data for a student model, the system comprising:

one or more computer processors; and

one or more computer readable storage media that store computer program instructions, wherein

the one or more computer processors are configured to execute the computer program instructions to:

receive an acoustic feature with one or more associated reference labels;

generate a first soft label for the acoustic feature based on a teacher model, wherein the first soft label is a probability prediction for the acoustic feature which places a word or a word segment in a specific class for which the teacher model is designed to classify;

identify one or more candidate soft labels based on each reference label of the one or more associated reference labels, wherein the identified one or more candidate soft labels are from a soft label pool;

determine if each of the identified one or more candidate soft labels is similar to the acoustic feature, wherein the identified one or more candidate soft labels is similar to the acoustic feature if the identified one or more candidate soft labels has the same sequence of phonemes as the acoustic feature;

generate a similarity score for each of the identified one or more candidate soft labels, wherein the similarity score is generated based on the determination that each of the identified one or more candidate soft labels is similar to the acoustic feature;

determine a second soft label, wherein

the identified one or more candidate soft labels include the second soft label, and

the similarity score of the second soft label is above a specific threshold; and

responsive to the determination of the second soft label:

output the second soft label;

train the student model using the generated first soft label and the second soft label, wherein the student model is an automatic speech recognition system; and

operate the trained student model as a speech recognition system on a user device.

7. The system of claim 6 , wherein

the determination if each of the identified one or more candidate soft labels is similar to the acoustic feature further comprises extraction, from the soft label pool, of a first word sequence, and

the extracted first word sequence has the same length as a second word sequence of the acoustic feature at a state level.

8. The system of claim 6 , wherein

the determination if each of the identified one or more candidate soft labels is similar to the acoustic feature further comprises extraction, from the soft label pool, of a first phoneme sequence, and

the extracted first phoneme sequence has the same length as a second phoneme sequence of the acoustic feature.

9. The system of claim 6 , wherein the one or more computer processors are further configured to execute the computer program instructions to generate feature vectors for a source code of a plurality of functions and a plurality of classes.

10. The system of claim 6 , wherein the teacher model is a visual geometry group model.

11. The system of claim 6 , wherein the student model is a convolutional neural network.

12. A computer program product for code refactor renaming comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processors to perform a function, the function comprising:

receive an acoustic feature with one or more associated reference labels;

generate a first soft label for the acoustic feature based on a teacher model, wherein the first soft label is a probability prediction for the acoustic feature which places a word or word segment in a specific class for which the teacher model is designed to classify;

identify one or more candidate soft labels based on each reference label of the one or more associated reference labels, wherein the identified one or more candidate soft labels are from a soft label pool;

determine if each of the identified one or more candidate soft labels is similar to the acoustic feature, wherein the identified one or more candidate soft labels is similar to the acoustic feature if the identified one or more candidate soft labels has the same sequence of phonemes as the acoustic feature;

generate a similarity score for each of the identified one or more candidate soft labels, wherein the similarity score is generated based on the determination that each of the identified one or more candidate soft labels is similar to the acoustic feature;

determine a second soft label, wherein

the identified one or more candidate soft labels include the second soft label, and

the similarity score of the second soft label is above a specific threshold; and

responsive to the determination of the second soft label:

output the second soft label;

train a student model using the generated first soft label and the second soft label, wherein the student model is an automatic speech recognition system; and

operate the trained student model as a speech recognition system on a user device.

13. The computer program product of claim 12 , wherein

the determination if each of the identified one or more candidate soft labels is similar to the acoustic feature includes extraction, from the soft label pool, of a first word sequence, and

the extracted first word sequence has the same length as a second word sequence of the acoustic feature at a state level.

14. The computer program product of claim 12 , wherein

the determination if each of the identified one or more candidate soft labels is similar to the acoustic feature includes extraction, from the soft label pool, of a first phoneme sequence, and

the extracted first phoneme sequence has the same length as a second phoneme sequence of the acoustic feature.

15. The computer program product of claim 12 , wherein the teacher model is a visual geometry group model.

16. The computer program product of claim 12 , wherein the student model is a convolutional neural network.

Assignments (1)

Continuity (1)

References Cited (22)