Systems and methods for out-of-distribution classification
An embodiment proposed herein uses sparsification techniques to train the neural network with a high feature dimension that may yield desirable in-domain detection accuracy but may prune away dimensions in the output that are less important. Specifically, a sparsification vector is generated based on Gaussian distribution (or other probabilistic distribution) and is used to multiply with the higher dimension output to reduce the number of feature dimensions. The pruned output may be then used for the neural network to learn the sparsification vector. In this way, out-of-distribution detection accuracy can be improved.
1. A system for training a neural network for out-of-distribution detection, the system comprising:
a communication interface that receives a plurality of training samples having a first feature dimension;
a memory containing machine readable medium storing machine executable code; and
one or more processors coupled to the memory and configurable to execute the machine executable code to cause the one or more processors to:
train the neural network using the plurality of training samples;
generate, via the neural network, a classification output in response to an input sample having the first feature dimension;
generate a pruned output by using a Gaussian distribution based sparsification vector to reduce a dimension of the classification output to a second feature dimension;
computing a loss based on the pruned output and the input sample; and
updating the sparsification vector by backpropagation based on the computed loss.
2. The system of claim 1 , wherein the one or more processors are configurable to execute the machine executable code to cause the one or more processors to train the neural network by obtaining a set of parameters for the neural network from the training.
3. The system of claim 2 , wherein the one or more processors are configurable to execute the machine executable code to cause the one or more processors to:
when the training is complete:
freeze the set of parameters of the neural network; and
update the sparsification vector by backpropagation based on the computed loss without modifying the set of parameters of the neural network.
4. The system of claim 1 , wherein the sparsification vector has a number of zero entries that set unused dimensions of the classification output to zero when multiplied with the classification output.
5. The system of claim 1 , wherein the one or more processors are configurable to execute the machine executable code to cause the one or more processors to compute the loss based on the pruned output and the input sample by:
computing a probability indicating a likelihood that the training sample is in-distribution or out-of-distribution by applying a softmax operation on the pruned output; and
computing a cross entropy loss of the probability.
6. A method for training a neural network for out-of-distribution detection, the method comprising:
receiving, via a communication interface, a plurality of training samples having a first feature dimension;
training the neural network using the plurality of training samples;
generating, via the neural network, a classification output in response to an input sample having the first feature dimension;
generating a pruned output by using a Gaussian distribution based sparsification vector to reduce a dimension of the classification output to a second feature dimension;
computing a loss based on the pruned output and the input sample; and
updating the sparsification vector by backpropagation based on the computed loss.
7. The method of claim 6 , further comprising obtaining a set of parameters for the neural network from the training.
8. The method of claim 7 , further comprising:
when the training is complete:
freezing the set of parameters of the neural network; and
updating the sparsification vector by backpropagation based on the computed loss without modifying the set of parameters of the neural network.
9. The method of claim 6 , wherein the sparsification vector has a number of zero entries that set unused dimensions of the classification output to zero when multiplied with the classification output.
10. The method of claim 6 , wherein the computing the loss based on the pruned output and the input sample comprises:
computing a probability indicating a likelihood that the training sample is in-distribution or out-of-distribution by applying a softmax operation on the pruned output; and
computing a cross entropy loss of the probability.