Granted Patent B2

US 12,524,622 · App. 17/557,219 · Granted Jan 13, 2026

Inventors: Pavan Buduguppa (Hyderabad, IN); Ramasubramanian Sundaram (Hyderabad, IN); Veera Raghavendra Elluru (Hyderabad, IN)

Assignee: Genesys Cloud Services, Inc.

G06F40/40G06F40/30G06N3/045G06N3/082G06N5/022

Abstract

Claims (63)

1 . A method for creating a student model from a teacher model for use in knowledge distillation, the method comprising the steps of:

providing a first model, wherein:

the first model comprises a neural network having a plurality of layers; and

each of the plurality of layers comprises neurons, each of the neurons having weights equal to a number of synapses leading to the neuron from each of the neurons in a preceding layer;

using a first instance of the first model to create the teacher model by training the first instance of the first model on a training dataset;

using a second instance of the first model to create the student model by training the second instance of the first model on a subset of the training dataset;

identifying corresponding layers in the teacher model and the student model, the corresponding layers comprising a trained layer in the teacher model and a trained layer in the student model derived from a same level in the first model;

for each of the corresponding layers:

determining values of the weights in the trained layer of the teacher model;

determining values of the weights in the trained layer of the student model; and

computing a weight similarity criterion, the weight similarity criterion comprising a value representing a degree of similarity between the values of the weights in the trained layer of the teacher model versus the values for the weights in the trained layer of the student model, wherein the weight similarity criterion comprises a distance between flattened values of the weights in the trained layer of the teacher model versus flattened values for the weights in the trained layer of the student model;

ranking the corresponding layers according to the weight similarity criterion;

selecting, based on the ranking of the corresponding layers according to the weight similarity criterion, one or more of the corresponding layers for designation as one or more discard layers, wherein the one or more discard layers comprise the one or more corresponding layers ranked as having the highest degree of similarity; and

modifying the student model by removing from the student model the one or more discard layers.

2 . The method of claim 1 ,

wherein the subset of the training dataset comprises less than 10% of the training dataset.

3 . The method of claim 1 , further comprising the step of training the modified student model so that an output of the modified student model over a transfer dataset mimics an output of the teacher model over the transfer dataset to within an acceptable loss of validity as defined by a predetermined loss function.

4 . The method of claim 3 , further comprising the step of employing the modified student model for a natural language processing task in a contact center, the natural language processing task comprising at least one of natural language inference, sentiment classification, and semantic textual similarity.

5 . The method of claim 4 , wherein the first model comprises a bidirectional encoder representations from transforms architecture.

6 . A system for creating a student model from a teacher model for use in knowledge distillation, the system comprising:

a processor; and

a memory, wherein the memory stores instructions that, when executed by the processor, cause the processor to perform the steps of:

providing a first model, wherein:

the first model comprises a neural network having a plurality of layers; and

each of the plurality of layers comprises neurons, each of the neurons having weights equal to a number of synapses leading to the neuron from each of the neurons in a preceding layer;

using a first instance of the first model to create the teacher model by training the first instance of the first model on a training dataset;

using a second instance of the first model to create the student model by training the second instance of the first model on a subset of the training dataset;

identifying corresponding layers in the teacher model and the student model, the corresponding layers comprising a trained layer in the teacher model and a trained layer in the student model derived from a same level in the first model;

for each of the corresponding layers:

determining values of the weights in the trained layer of the teacher model;

determining values of the weights in the trained layer of the student model; and

computing a weight similarity criterion, the weight similarity criterion comprising a value representing a degree of similarity between the values of the weights in the trained layer of the teacher model versus the values for the weights in the trained layer of the student model, wherein the weight similarity criterion comprises a distance between flattened values of the weights in the trained layer of the teacher model versus flattened values for the weights in the trained layer of the student model;

ranking the corresponding layers according to the weight similarity criterion;

selecting, based on the ranking of the corresponding layers according to the weight similarity criterion, one or more of the corresponding layers for designation as one or more discard layers, wherein the one or more discard layers comprise the one or more corresponding layers ranked as having the highest degree of similarity; and

modifying the student model by removing from the student model the one or more discard layers.

7 . The system of claim 6 ,

wherein the subset of the training dataset comprises less than 10% of the training dataset.

8 . The system of claim 6 , wherein the memory further stores instructions that, when executed by the processor, cause the processor to perform the step of:

training the modified student model so that an output of the modified student model over a transfer dataset mimics an output of the teacher model over the transfer dataset to within an acceptable loss of validity as defined by a predetermined loss function.

9 . The system of claim 8 , wherein the memory further stores instructions that, when executed by the processor, cause the processor to perform the step of:

employing the modified student model for a natural language processing task in a contact center, the natural language processing task comprising at least one of natural language inference, sentiment classification, and semantic textual similarity.

10 . The system of claim 9 , wherein the first model comprises a bidirectional encoder representations from transforms architecture.

11 . A non-transitory computer-readable storage medium having computer instructions stored therein, which when executed by a computer, cause the computer to perform a method for creating a student model from a teacher model for use in knowledge distillation, the method comprising the steps of:

providing a first model, wherein:

the first model comprises a neural network having a plurality of layers; and

each of the plurality of layers comprises neurons, each of the neurons having weights equal to a number of synapses leading to the neuron from each of the neurons in a preceding layer;

using a first instance of the first model to create the teacher model by training the first instance of the first model on a training dataset;

using a second instance of the first model to create the student model by training the second instance of the first model on a subset of the training dataset;

identifying corresponding layers in the teacher model and the student model, the corresponding layers comprising a trained layer in the teacher model and a trained layer in the student model derived from a same level in the first model;

for each of the corresponding layers:

determining values of the weights in the trained layer of the teacher model;

determining values of the weights in the trained layer of the student model; and

computing a weight similarity criterion, the weight similarity criterion comprising a value representing a degree of similarity between the values of the weights in the trained layer of the teacher model versus the values for the weights in the trained layer of the student model, wherein the weight similarity criterion comprises a distance between flattened values of the weights in the trained layer of the teacher model versus flattened values for the weights in the trained layer of the student model;

ranking the corresponding layers according to the weight similarity criterion;

selecting, based on the ranking of the corresponding layers according to the weight similarity criterion, one or more of the corresponding layers for designation as one or more discard layers, wherein the one or more discard layers comprise the one or more corresponding layers ranked as having the highest degree of similarity; and

modifying the student model by removing from the student model the one or more discard layers.

12 . The non-transitory computer-readable storage medium of claim 11 ,

wherein the subset of the training dataset comprises less than 10% of the training dataset.

13 . The non-transitory computer-readable storage medium of claim 11 , wherein the computer instructions stored therein, when executed by a computer, cause the computer to further perform the step of:

training the modified student model so that an output of the modified student model over a transfer dataset mimics an output of the teacher model over the transfer dataset to within an acceptable loss of validity as defined by a predetermined loss function.

14 . The non-transitory computer-readable storage medium of claim 13 , wherein the computer instructions stored therein, when executed by a computer, cause the computer to further perform the step of:

employing the modified student model for a natural language processing task in a contact center, the natural language processing task comprising at least one of natural language inference, sentiment classification, and semantic textual similarity.

15 . The non-transitory computer-readable storage medium of claim 14 , wherein the first model comprises a bidirectional encoder representations from transforms architecture.

Assignments (3)

Continuity (1)

References Cited (11)