Systems and methods for a k-nearest neighbor based mechanism of natural language processing models
Embodiments described herein adopts a k nearest neighbor (kNN) mechanism over a model's hidden representations to identify training examples closest to a given test example. Specifically, a training set of sequences and a test sequence are received, each of which is mapped to a respective hidden representation vector using a base model. A set of indices for each sequence index that minimizes a distance between the respective hidden state vector and a test hidden state vector is then determined A weighted k-nearest neighbor probability score can then be computed from the set of indices to generate a probability distribution over labels for the test sequence.
1. A method of identifying inaccurate labels in training data, the method comprising:
receiving, at a data interface, a training set of sequences, wherein each sequence from the training set is paired with a target label from a plurality of labels;
mapping, by a neural network, each training sequence to a respective normalized hidden representation vector, including:
computing the respective normalized hidden representation vector based on a dataset-wise batch normalization of hidden representation vectors with a mean and a standard deviation over hidden states of the neural network responsive to the training set of sequences,
wherein the mean and the standard deviation of hidden states are obtained over the training set of sequences, and
wherein the computing further comprises dividing a difference between a first hidden representation vector and the mean by a sum of the standard deviation and a numerical stability parameter;
receiving a testing sequence at inference stage;
after receiving the testing sequence at inference stage, mapping, by the neural network, the test sequence to a normalized test hidden representation vector;
determining, among the training set of sequences, a set of sequence indices that lead to a set of smallest distances between respective normalized respective hidden state vectors and the normalized test hidden state vector;
computing, for each training sequence in the set of sequence indices, a weighted probability score based on a set of distances corresponding to the set of sequence indices;
generate a probability distribution over the plurality of labels for the test sequence based on computed weighted probability scores and one-hot encodings of each target label in the plurality of labels; and
identifying a mislabeled training sequence from the training set of sequences when a direct prediction by the neural network is different from a prediction based on the generated probability distribution.
2. The method of claim 1 , further comprising:
caching the mapped respective normalized hidden representation vectors at a memory;
and querying the cached mapped respective normalized hidden representation vectors for determining the set of sequence indices and comparing with the normalized test hidden state vector.
3. The method of claim 1 , further comprising generating a classifier prediction for the test sequence based on the generated probability distribution.
4. The method of claim 1 , wherein the respective normalized hidden representation vector is a representation corresponding to a specific token that is added at a beginning of the respective sequence in a last layer of the neural network.
5. The method of claim 1 , further comprising:
generating, by the neural network, a prediction in response to the testing sequence and a confidence score associated with the predicted distribution; and
in response to determining that the confidence score is lower than a threshold, computing the weighted probability score for fine-tuning the prediction of the testing sequence.
6. The method of claim 1 , wherein the weighted probability score for each training sequence in the set of sequence indices is computed using a softmax operation over negatives of the set of smallest distances.
7. A system for identifying inaccurate labels in training data, the system comprising:
a data interface that receives a training set of sequences, wherein each sequence from the training set is paired with a target label from a plurality of labels;
a memory that stores a neural model; and
one or more hardware processors that:
map, by the neural model, each training sequence to a respective normalized hidden representation vector, including:
computing the respective normalized hidden representation vector based on a dataset-wise batch normalization of hidden representation vectors with a mean and a standard deviation over hidden states of the neural network responsive to the training set of sequences,
wherein the mean and the standard deviation of hidden states are obtained over the training set of sequences, and
wherein the computing further comprises dividing a difference between a first hidden representation vector and the mean by a sum of the standard deviation and a numerical stability parameter;
in response to receiving a testing sequence at inference stage, map, by the neural model, the test sequence to a normalized test hidden representation vector;
determine, among the training set of sequences, a set of sequence indices that lead to a set of smallest distances between respective normalized respective hidden state vectors and the normalized test hidden state vector;
compute, for each training sequence in the set of sequence indices, a weighted probability score based on a set of distances corresponding to the set of sequence indices;
generate a probability distribution over the plurality of labels for the test sequence based on the weighted probability score and one-hot encodings of each target label in the plurality of labels; and
identifying a mislabeled training sequence from the training set of sequences when a direct prediction by the neural model is different from a prediction based on the generated probability distribution.
8. The system of claim 7 , wherein the one or more hardware processors further:
cache the mapped respective normalized hidden representation vectors at a memory; and
query the cached mapped respective normalized hidden representation vectors for determining the set of sequence indices and comparing with the normalized test hidden state vector.
9. The system of claim 7 , wherein the one or more hardware processors further generate a classifier prediction for the test sequence based on the generated probability distribution.
10. The system of claim 7 , wherein the respective normalized hidden representation vector is a representation corresponding to a specific token that is added at a beginning of the respective sequence in a last layer of the neural network.
11. The system of claim 7 , wherein the one or more hardware processors further:
generate, by the neural network, a prediction in response to the testing sequence and a confidence score associated with the predicted distribution; and
in response to determining that the confidence score is lower than a threshold, compute the weighted probability score for fine-tuning the prediction of the testing sequence.
12. A non-transitory processor-readable medium storing a plurality of instructions for identifying inaccurate labels in training data, the instructions being processor-executed by one or more hardware processors to perform operations comprising:
receiving, at a data interface, a training set of sequences, wherein each sequence from the training set is paired with a target label from a plurality of labels;
mapping, by a neural network, each training sequence to a respective normalized hidden representation vector, including:
computing the respective normalized hidden representation vector based on a dataset-wise batch normalization of hidden representation vectors with a mean and a standard deviation over hidden states of the neural network responsive to the training set of sequences,
wherein the mean and the standard deviation of hidden states are obtained over the training set of sequences, and
wherein the computing further comprises dividing a difference between a first hidden representation vector and the mean by a sum of the standard deviation and a numerical stability parameter;
in response to receiving a testing sequence at inference stage, mapping, by the neural network, the test sequence to a normalized test hidden representation vector;
determining, among the training set of sequences, a set of sequence indices that lead to a set of smallest distances between respective normalized respective hidden state vectors and the normalized test hidden state vector;
computing, for each training sequence in the set of sequence indices, a weighted probability score based on a set of distances corresponding to the set of sequence indices;
generate a probability distribution over the plurality of labels for the test sequence based on the weighted probability score and one-hot encodings of each target label in the plurality of labels; and
identifying a mislabeled training sequence from the training set of sequences when a direct prediction by the neural network is different from a prediction based on the generated probability distribution.
13. The medium of claim 12 , wherein the operations further comprise:
caching the mapped respective normalized hidden representation vectors at a memory;
and querying the cached mapped respective normalized hidden representation vectors for determining the set of sequence indices and comparing with the normalized test hidden state vector.
14. The medium of claim 12 , wherein the operations further comprise:
generating, by the neural network, a prediction in response to the testing sequence and a confidence score associated with the predicted distribution; and
in response to determining that the confidence score is lower than a threshold, computing the weighted probability score for fine-tuning the prediction of the testing sequence.