Accelerating inference of neural network models via dynamic early exits
Methods, systems, and computer program products for accelerating inference of neural network models via dynamic early exits are provided herein. A computer-implemented method includes determining a plurality of candidate exit points of a neural network model; obtaining a plurality of outputs of the neural network model for data samples in a target dataset, wherein the plurality of outputs comprises early outputs of the neural network model from the plurality of candidate exit points and regular outputs of the neural network model; and a set of one or more exit points from the plurality of candidate exits points that are dependent on the target dataset based at least in part on the plurality of outputs.
1 . A computer-implemented method, the method comprising:
determining a plurality of candidate exit points associated with respective layers of a neural network model;
processing, using the neural network model, one or more data samples in a target dataset to obtain a plurality of outputs, wherein the plurality of outputs comprises a first set of outputs of the neural network model from respective ones of the plurality of candidate exit points and a second set of outputs of the neural network model from a final layer of the neural network model;
storing the first set of outputs and the second set of outputs in one or more memory layers, integrated into an architecture of the neural network model;
performing a comparative analysis between the first set and the second set of outputs;
selecting a set of one or more exit points from the plurality of candidate exit points that are dependent on the target dataset based at least in part on the comparative analysis of the first set of outputs and the second set of outputs stored in the one or more memory layers of the neural network model, wherein selecting the set of one or more exit points is based at least in part on determination that two or more candidate exit points of the plurality of candidate exit points are contiguous exit points in the neural network model, wherein a first one of the contiguous exit points is associated with a first layer of the neural network model that provides an output directly to a second layer of the neural network model associated with a second one of the contiguous exit points; and
processing, using the neural network model, one or more additional data samples in the target dataset based at least in part on the selected set of one or more exit points;
wherein the method is carried out by at least one computing device.
2 . The computer-implemented method of claim 1 , wherein selecting the set of one or more exit points comprises computing a set of statistics based on the first set of outputs and the second set of outputs.
3 . The computer-implemented method of claim 2 , wherein computing the set of statistics comprises at least one of:
applying a KL divergence technique;
applying a cross entropy technique; and
determining whether at least one output in the first set is different than the at least one output in the second set for given ones of the data samples.
4 . The computer-implemented method of claim 2 , wherein selecting the set of one or more exit points comprises ranking the plurality of candidate exit points based on the set of statistics.
5 . The computer-implemented method of claim 1 , wherein selecting the set of one or more exit points comprises determining a threshold exit value for each exit point in the set.
6 . The computer-implemented method of claim 1 , wherein selecting the set of one or more exit points comprises training the neural network model using a modified loss function.
7 . The computer-implemented method of claim 6 , wherein training the neural network model is based on one or more user inputs.
8 . The computer-implemented method of claim 6 , wherein selecting the set of one or more exit points is based at least in part on a combination of exit-wise losses determined by the modified loss function.
9 . The computer-implemented method of claim 7 , wherein the one or more user inputs comprise at least one of:
a number of exits to be used in the neural network model;
a threshold change in accuracy of the neural network model;
one or more constraints on placement of the exits; and
one or more computing constraints.
10 . A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to:
determine a plurality of candidate exit points associated with respective layers of a neural network model;
process, using the neural network model, one or more data samples in a target dataset to obtain a plurality of outputs, wherein the plurality of outputs comprises a first set of outputs of the neural network model from respective ones of the plurality of candidate exit points and a second set of outputs of the neural network model from a final layer of the neural network model;
store the first set of outputs and the second set of outputs in one or more memory layers integrated into an architecture of the neural network model;
perform a comparative analysis between the first set and the second set of outputs;
select a set of one or more exit points from the plurality of candidate exit points that are dependent on the target dataset based at least in part on the comparative analysis of the first set of outputs and the second set of outputs stored in the one or more memory layers of the neural network model, wherein selecting the set of one or more exit points is based at least in part on determination that two or more candidate exit points of the plurality of candidate exit points are contiguous exit points in the neural network model, wherein a first one of the contiguous exit points is associated with a first layer of the neural network model that provides an output directly to a second layer of the neural network model associated with a second one of the contiguous exit points; and
process, using the neural network model, one or more additional data samples in the target dataset based at least in part on the selected set of one or more exit points.
11 . The computer program product of claim 10 , wherein selecting the set of one or more exit points comprises computing a set of statistics based on the first set of outputs and the second set of outputs.
12 . The computer program product of claim 11 , wherein computing the set of statistics comprises at least one of:
applying a KL divergence technique;
applying a cross entropy technique; and
determining whether at least one output in the first set is different than the at least one output in the second set for given ones of the data samples.
13 . The computer program product of claim 11 , wherein selecting the set of one or more exit points comprises ranking the plurality of candidate exit points based on the set of statistics.
14 . The computer program product of claim 10 , wherein selecting the set of one or more exit points comprises determining a threshold exit value for each exit point in the set.
15 . The computer program product of claim 10 , wherein selecting the set of one or more exit points comprises training the neural network model using a modified loss function.
16 . The computer program product of claim 15 , wherein training the neural network model is based on one or more user inputs.
17 . The computer program product of claim 15 , wherein selecting the set of one or more exit points is based at least in part on a combination of exit-wise losses determined by the modified loss function.
18 . The computer program product of claim 16 , wherein the one or more user inputs comprise at least one of:
a number of exits to be used in the neural network model;
a threshold change in accuracy of the neural network model;
one or more constraints on placement of the exits; and
one or more computing constraints.
19 . A system comprising:
a memory configured to store program instructions;
a processor operatively coupled to the memory to execute the program instructions to:
determine a plurality of candidate exit points associated with respective layers of a neural network model;
process, using the neural network model, one or more data samples in a target dataset to obtain a plurality of outputs, wherein the plurality of outputs comprises a first set of outputs of the neural network model from respective ones of the plurality of candidate exit points and a second set of outputs of the neural network model from a final layer of the neural network model;
store the first set of outputs and the second set of outputs in one or more memory layers integrated into an architecture of the neural network model;
perform a comparative analysis between the first set and the second set of outputs;
select a set of one or more exit points from the plurality of candidate exit points that are dependent on the target dataset based at least in part on the comparative analysis of the first set of outputs and the second set of outputs stored in the one or more memory layers of the neural network model, wherein selecting the set of one or more exit points is based at least in part on determination that two or more candidate exit points of the plurality of candidate exit points are contiguous exit points in the neural network model, wherein a first one of the contiguous exit points is associated with a first layer of the neural network model that provides an output directly to a second layer of the neural network model associated with a second one of the contiguous exit points; and
process, using the neural network model, one or more additional data samples in the target dataset based at least in part on the selected set of one or more exit points.
20 . The computer-implemented method of claim 1 , wherein the set of one or more exit points is selected based at least on part on whether at least one candidate of the plurality of candidate exit points is located in a second half of the layers of the neural network model.