Systems and methods for out-of-distribution classification
An embodiment provided herein preprocesses the input samples to the classification neural network, e.g., by adding Gaussian noise to word/sentence representations to make the function of the neural network satisfy Lipschitz property such that a small change in the input does not cause much change to the output if the input sample is in-distribution. Method to induce properties in the feature representation of neural network such that for out-of-distribution examples the feature representation magnitude is either close to zero or the feature representation is orthogonal to all class representations. Method to generate examples that are structurally similar to in-domain and semantically out-of domain for use in out-of-domain classification training. Method to prune feature representation dimension to mitigate long tail error of unused dimension in out-of-domain classification. Using these techniques, the accuracy of both in-domain and out-of-distribution identification can be improved.
1. A system for identifying out-of-distribution data samples using a neural network, the system comprising:
a communication interface that receives an input sample;
a memory containing machine readable medium storing machine executable code; and
one or more processors coupled to the memory and configurable to execute the machine executable code to cause the one or more processors to:
generate a vector representation of the input sample;
preprocess the vector representation of the input sample by adding a Gaussian noise component having a variance value to the vector representation that yields an input-to-output function Lipschitz tight;
generate an output via the neural network from the preprocessed input sample; and
determine whether the input sample is in-distribution or out-of-distribution based on the generated output.
2. The system of claim 1 , wherein the one or more processors are configurable to execute the machine executable code to cause the one or more processors to:
add the Gaussian noise component to each vector representation of a respective word within the input sample.
3. The system of claim 1 , wherein the one or more processors are configurable to execute the machine executable code to cause the one or more processors to:
add the Gaussian noise component at a sentence level of the input sample.
4. The system of claim 1 , wherein the one or more processors are configurable to execute the machine executable code to cause the one or more processors to:
select the variance value such that after the preprocessing with the Gaussian noise component, a resulting function corresponding to the neural network satisfies a Lipschitz condition.
5. The system of claim 1 , wherein the one or more processors are configurable to execute the machine executable code to cause the one or more processors to:
train the neural network using a loss computed from the classification output generated from the preprocessed input sample.
6. A method for identifying out-of-distribution data samples using a neural network, the method comprising:
receiving, via a communication interface, an input sample;
generating a vector representation of the input sample;
preprocessing the vector representation of the input sample by adding a Gaussian noise component having a variance value to the vector representation that yields an input-to-output function Lipschitz tight;
generating an output via the neural network from the preprocessed input sample; and
determining whether the input sample is in-distribution or out-of-distribution based on the generated output.
7. The method of claim 6 , further comprising:
adding the Gaussian noise component to each vector representation of a respective word within the input sample.
8. The method of claim 6 , further comprising:
adding the Gaussian noise component at a sentence level of the input sample.
9. The method of claim 6 , further comprising:
selecting the variance value such that after the preprocessing with the Gaussian noise component, a resulting function corresponding to the neural network satisfies a Lipschitz condition.
10. The method of claim 6 , further comprising:
training the neural network using a loss computed from the classification output generated from the preprocessed input sample.
11. A method for identifying out-of-distribution data samples using a neural network, the method comprising:
receiving, via a communication interface, an input sample;
generating, at a penultimate layer of the neural network, a vector representation of the input sample;
retrieving a number of reference class vectors corresponding to a number of pre-defined classes;
comparing the vector representation of the input sample against the number of reference class vectors; and
determining whether the input sample is in-distribution or out-of-distribution based on whether the vector representation of the input sample is orthogonal to the number of reference class vectors based on the comparison.
12. The method of claim 11 , further comprising:
determining whether the input sample is in-distribution or out-of-distribution based on whether the vector representation of the input sample is within a threshold distance to an origin of a multi-dimensional space hosting the number of reference class vectors.
13. The method of claim 11 , wherein the comparing the vector representation of the input sample against the number of reference class vectors comprises:
multiplying a kernel vector of weights to the vector representation of the input sample from the penultimate layer.
14. The method of claim 13 , further comprising:
computing a minimum angle made by the vector representation against one of the number of reference class vectors by maximizing a distance of a vector product from an origin.
15. The method of claim 14 , further comprising:
converting the maximized distance of the vector product from the multiplication to a probability value indicating a likelihood that the input sample is in-distribution or out-of-distribution.
16. The method of claim 15 , wherein the converting the maximized distance of the vector product further comprises computing a tangent value of the vector product when the distance of the vector product to the origin is maximized.
17. The method of claim 13 , wherein the input sample is a training sample having a pre-defined classifier, and the method further comprises:
computing a loss based on the determination of whether the input sample is in-distribution or out-of-distribution and the pre-defined classifier; and
updating parameters associated with the penultimate layer of the neural network and the kernel vector of weights via backpropagation based on the loss.
18. A system for identifying out-of-distribution data samples using a neural network, the system comprising:
a communication interface that receives an input sample;
a memory containing machine readable medium storing machine executable code; and
one or more processors coupled to the memory and configurable to execute the machine executable code to cause the one or more processors to:
generate, at a penultimate layer of the neural network, a vector representation of the input sample;
retrieve a number of reference class vectors corresponding to a number of pre-defined classes;
compare the vector representation of the input sample against the number of reference class vectors; and
determine whether the input sample is in-distribution or out-of-distribution based on whether the vector representation of the input sample is orthogonal to the number of reference class vectors based on the comparison.
19. The system of claim 18 , wherein the one or more processors are configurable to execute the machine executable code to cause the one or more processors to:
determine whether the input sample is in-distribution or out-of-distribution based on whether the vector representation of the input sample is within a threshold distance to an origin of a multi-dimensional space hosting the number of reference class vectors.
20. The system of claim 18 , wherein the one or more processors are configurable to execute the machine executable code to cause the one or more processors to compare the vector representation of the input sample against the number of reference class vectors by:
multiplying a kernel vector of weights to the vector representation of the input sample from the penultimate layer;
computing a minimum angle made by the vector representation against one of the number of reference class vectors by maximizing a distance of a vector product from an origin; and
converting the maximized distance of the vector product from the multiplication to a probability value indicating a likelihood that the input sample is in-distribution or out-of-distribution.