Distributed sample selection with self-labeling
In some embodiments, techniques for self-labeling to extract a representative set of samples from a large-scale set of unlabeled documents (e.g., a set that represents a distribution of the large-scale set) are provided. The samples of the representative set may then be used to classify the documents of the large-scale set.
1 . A computer-implemented method, comprising using at least one processor to perform the following operations:
receiving, by a feature representation network, a plurality of samples, wherein each sample is associated with an unlabeled document devoid of a ground truth label;
generating, using the feature representation network and for each of the plurality of samples, a corresponding one of a plurality of feature representations;
obtaining neighborhood information from the plurality of feature representations, wherein neighborhood information is computed in a learned embedding space produced by the feature representation network and indicates, for each sample, a set of nearest neighbors between samples based on a distance in the learned embedding space;
using a clustering network that is trained using the plurality of feature representations and the neighborhood information to generate, for the plurality of samples, a corresponding plurality of cluster predictions;
selecting a set of confident samples using the plurality of cluster predictions and the neighborhood information, wherein a sample of the plurality of samples is included in the set of confident samples when:
a predicted probability for a cluster assignment taken as a maximum element of a predictive cluster probability vector of the corresponding plurality of cluster predictions associated with the sample satisfies a probability threshold; and
a proportion of neighbors of the set of nearest neighbors sharing a same cluster assignment associated with the sample satisfies a consistency threshold;
using a classifier model that is trained using the set of confident samples to generate, for the plurality of samples, a corresponding plurality of self-labels, wherein each of the plurality of self-labels indicates a cluster assignment of a corresponding sample of the plurality of samples; and
selecting a set of representative samples from among the plurality of samples, of self label, wherein the set of representative samples comprises a total number of samples including a number of confident samples selected from the set of confident samples and a number of non-confident samples, and wherein the total number of samples is determined using a configurable parameter that:
scales the total number of samples as a proportion of a size of the plurality of samples; and
defines a ratio of the number of confident samples to the number of non-confident samples to be within a predefined range.
2 . The computer-implemented method according to claim 1 , wherein the feature representation network is trained using a contrastive learning loss function.
3 . The computer-implemented method according to claim 1 , wherein obtaining neighborhood information comprises:
calculating, for a sample of the plurality of samples, a number d of samples of the plurality of samples whose feature representations are closest to the feature representation of the sample, where d is a hyper-parameter identified based on a trial and error method, and wherein the distance comprises a Euclidean distance.
4 . The computer-implemented method according to claim 1 , wherein the feature representation network includes a backbone network and a feature block, and wherein generating the plurality of feature representations comprises:
converting each unlabeled document into a respective input vector;
transforming, using the backbone network, each respective input vector into a respective feature vector; and
dimensionally reducing, using the feature block, each respective feature vector into a respective feature representation of the plurality of feature representations.
5 . The computer-implemented method according to claim 1 , wherein the predicted cluster probability vector comprises a length, m, where m is a number of clusters.
6 . The computer-implemented method according to claim 1 , wherein the probability threshold for the cluster assignment of the corresponding plurality of cluster predictions for each sample is defined on a sample level basis and is implemented as a tunable hyper-parameter.
7 . The computer-implemented method according to claim 1 , wherein the consistency threshold for the proportion of neighbors sharing the same cluster assignment associated with each sample is defined on a community level basis and implemented as a tunable hyper-parameter.
8 . The computer-implemented method according to claim 1 , wherein the set of representative samples approximates a distribution of data within the plurality of samples.
9 . The computer-implemented method according to claim 1 , wherein the set of representative samples includes samples from each of a plurality of clusters indicated by the plurality of self-labels.
10 . The computer-implemented method according to claim 1 , wherein the configurable parameter further defines a proportion of the number of confident samples to the number of non-confident samples to be within a predefined range on a per-cluster basis.
11 . A computer-implemented method, comprising using at least one processor to perform the following operations in an ordered plurality of levels, including a first level and at least one subsequent level including a final level:
receiving, by a chunking module, a set of samples, wherein each sample is associated with an unlabeled document devoid of a ground truth label;
dividing, by the chunking module, the set of samples into a plurality of non-overlapping chunks of samples;
at the first level of the ordered plurality of levels, producing a plurality of sets of representative samples of the first level, comprising:
for each chunk of the plurality of non-overlapping chunks:
obtaining neighborhood information from a plurality of feature representations of each sample of each chunk, wherein neighborhood information is computed in a learned embedding space and indicates, for each sample, a set of nearest neighbors between samples based on a distance in the learned embedding space;
generating, for the samples of the chunk, a corresponding plurality of cluster predictions; and
selecting a plurality of confident samples from each cluster of the plurality of clusters to produce a corresponding one of the plurality of sets of representative samples of the first level, wherein a sample of the set of samples is included in the plurality of confident samples when:
a predicted probability for a cluster assignment taken as a maximum element of a predictive cluster probability vector of the corresponding plurality of cluster predictions associated with the sample satisfies a probability threshold; and
a proportion of neighbors of the set of nearest neighbors sharing a same cluster assignment associated with the sample satisfies a consistency threshold;
at each subsequent level of the ordered plurality of levels, producing at least one set of representative samples of the level, comprising:
pooling the plurality of sets of representative samples of the previous level among the ordered plurality of levels to obtain at least one pool, comprising at least one instance of:
pooling, by at least one pooling module, at least two of the plurality of sets of representative samples of a previous level to obtain a corresponding pool of the at least one pool; and
for each pool of the at least one pool assigning the samples of the pool among a plurality of clusters based on neighborhood information associated with the samples of the pool; and
selecting a plurality of samples from each cluster of the plurality of clusters to produce a corresponding one of at least one set of representative samples of the level, wherein the corresponding one of the at least one set of representative samples comprises a total number of samples including a number of confident samples selected from the plurality of confident samples and a number of non-confident samples, and wherein the total number of samples is determined using a configurable parameter that:
scales the total number of samples as a proportion of a size of the set of samples; and
defines a ratio of the number of confident samples to the number of non-confident samples to be within a predefined range;
training, by a global classifier training module, a global classifier model based on samples from the at least one set of representative samples of the final level of the ordered plurality of levels to obtain a trained global classifier; and
labeling, using the trained global classifier model, samples of the set of samples.
12 . The computer-implemented method according to claim 11 , wherein training the global classifier model includes using annotations of the samples from the at least one set of representative samples of the final level as ground truth labels.
13 . The computer-implemented method according to claim 12 , wherein the annotations of the samples are provided by at least one human.
14 . The computer-implemented method according to claim 12 , comprising determining that an accuracy of the trained global classifier model on the at least one set of representative samples of the final level satisfies a threshold.
15 . The computer-implemented method according to claim 11 , wherein the configurable parameter further defines a proportion of the number of confident samples to the number of non-confident samples to be within a predefined range on a per-cluster basis.
16 . A computer-implemented method to obtain a set of documents that is representative of a set of documents, the method comprising:
receiving a set of unlabeled documents, wherein each unlabeled document is devoid of a ground truth label;
at a first level of an ordered plurality of levels, dividing a set of unlabeled documents into a plurality of non-overlapping chunks;
for each chunk of the plurality of non-overlapping chunks:
training a corresponding instance of a clustering model on a training set of documents of the chunk;
obtaining neighborhood information from a plurality of feature representations of each unlabeled document of the chunk, wherein neighborhood information is computed in a learned embedding space and indicates, for each sample, a set of nearest neighbors between unlabeled documents based on a distance in the learned embedding space;
generating, for the unlabeled documents of the chunk, and using the trained corresponding instance, a corresponding plurality of cluster predictions; and
selecting a set of documents of the chunk that includes, for each of the plurality of clusters, a plurality of confident documents and a plurality of non-confident documents, wherein a document of the set of unlabeled documents is included in the plurality of confident documents when:
a predicted probability for a cluster assignment taken as a maximum element of a predictive cluster probability vector of the corresponding plurality of cluster predictions associated with the sample satisfies a probability threshold; and
a proportion of neighbors of the set of nearest neighbors sharing a same cluster assignment associated with the sample satisfies a consistency threshold;
at each of a second and subsequent levels of the ordered plurality of levels:
pooling sets of documents from a previous level to obtain at least one pool; and
for each of the at least one pool:
training a corresponding instance of a clustering model on a training set of documents of the pool;
clustering the documents of the pool, using the trained corresponding instance, into a plurality of clusters based on neighborhood information associated with the documents of the pool, the neighborhood information indicating a distance between the documents of the pool in an embedding space; and
selecting a set of documents of the pool that includes, for each of the plurality of clusters, a plurality of confident documents and a plurality of non-confident documents, wherein the set of documents comprises a total number of documents including a number of confident documents selected from the plurality of confident documents and a number of non-confident documents, and wherein the total number of documents is determined using a configurable parameter that:
scales the total number of documents as a proportion of a size of the set of unlabeled documents; and
defines a ratio of the number of confident documents to the number of non-confident documents to be within a predefined range.
17 . The computer-implemented method according to claim 16 , wherein the training set of documents comprise annotations of the documents of the training set of documents that are usable as ground truth labels.
18 . The computer-implemented method according to claim 17 , wherein the annotations of the documents are provided by at least one human.
19 . The computer-implemented method according to claim 16 , further comprising:
training a global classifier model based on documents from at least one set of representative documents of a final level of the ordered plurality of levels to obtain a trained global classifier, wherein the at least one set of representative documents comprises the plurality of confident documents and the plurality of non-confident documents; and
labeling, using the trained global classifier model, samples of the set of unlabeled documents.
20 . The computer-implemented method according to claim 16 , wherein the predicted cluster probability vector comprises a length, m, where m is an integer in a range of five to one hundred, wherein the probability threshold is at least 0.8 and the consistency threshold is at least 60%.