Anomalous text detection and entity identification using exploration-exploitation and pre-trained language models
There is a need for more effective and efficient anomalous text detection. This need can be addressed by, for example, solutions for anomalous text detection that include the steps of performing a group of exploration-exploitation keyword extraction iterations based at least in part on one or more training corpus data entries until a per-iteration keyword list for an ultimate exploration-exploitation keyword extraction iteration satisfies a keyword list threshold condition; and subsequent to performing the exploration-exploitation keyword extraction iterations: processing one or more input corpus data entries using the language-model-based binary classification model to generate one or more inferred anomaly probabilities, processing the one or more input corpus data entries using the keyword model to generate explanatory metadata for the one or more inferred anomaly probabilities, and performing one or more prediction-based actions based at least in part on the one or more inferred anomaly probabilities and the explanatory metadata.
1. A computer-implemented method comprising:
receiving, by one or more processors and via a plurality of exploration-exploitation keyword extraction iterations, a keyword list (i) based at least in part on a plurality of training corpus data entries and (ii) that satisfies a keyword list threshold condition, wherein an exploration-exploitation keyword extraction iteration of the plurality of exploration-exploitation keyword extraction iterations comprises:
(i) modifying the plurality of training corpus data entries by generating one or more anomalous data entries based at least in part on a prior per-iteration keyword list of a prior exploration-exploitation keyword extraction iteration that temporally precedes the exploration-exploitation keyword extraction iteration,
(ii) training a language-model-based binary classification model based at least in part on the one or more anomalous data entries, wherein the language-model-based binary classification model comprises:
(a) a bidirectional encoder layer that is configured to process the one or more anomalous data entries to generate an encoded representation,
(b) a dropout layer that is configured to process the encoded representation to generate a dropout representation,
(c) a fully-connected layer that is configured to process the dropout representation to generate a fully connected output; and
(d) a softmax layer that is configured to process the fully connected output to generate an anomaly probability for the one or more anomalous data entries,
(iii) inputting the one or more anomalous data entries to the language-model-based binary classification model to generate one or more per-entry anomaly probabilities, and
(iv) generating a per-iteration keyword list for the exploration-exploitation keyword extraction iteration based at least in part on the one or more per-entry anomaly probabilities, wherein the keyword list comprises the per-iteration keyword list for an ultimate exploration-exploitation keyword extraction iteration of the plurality of exploration-exploitation keyword extraction iterations; and
updating, by the one or more processors, a keyword model associated with the language-model-based binary classification model based at least in part on the keyword list, wherein the keyword model is configured to provide explanatory metadata for one or more inferred anomaly probabilities output by the language-model-based binary classification model.
2. The computer-implemented method of claim 1 , wherein the prior exploration-exploitation keyword extraction iteration is an initial exploration-exploitation keyword extraction iteration of the plurality of exploration-exploitation keyword extraction iterations and the one or more anomalous data entries of the initial exploration- exploitation keyword extraction iteration are generated by removing randomly-selected removable words from one or more training corpus data entries of the plurality of training corpus data entries.
3. The computer-implemented method of claim 1 , wherein updating the keyword model comprises:
inputting a keyword of the keyword of the keyword list to the bidirectional encoder layer to generate a per-keyword encoded representation of the keyword; and
generating, using a label-generating machine learning model, a labeled keyword cluster based at least in part on the per-keyword encoded representation.
4. The computer-implemented method of claim 3 , wherein generating the labeled keyword cluster comprises:
generating, using a principal component analysis layer, a reduced representation of the per-keyword encoded representation;
generating, using a t-distributed stochastic neighbor embedding layer, an unlabeled keyword cluster for the per-keyword encoded representation based at least in part on the reduced representation; and
generating, using a K-means layer, the labeled keyword cluster for the per-keyword encoded representation based on the unlabeled keyword cluster.
5. The computer-implemented method of claim 1 , wherein the keyword list threshold condition is satisfied when a deviation measure between the per-iteration keyword list and a prior per-iteration keyword list of the prior exploration-exploitation keyword extraction iteration satisfies a threshold deviation measure.
6. The computer-implemented method of claim 1 , wherein the keyword list threshold condition is satisfied when a keyword count for the per-iteration keyword list satisfies a threshold keyword count measure.
7. The computer-implemented method of claim 1 , further comprising:
inputting one or more input corpus data entries to the language-model-based binary classification model to generate the one or more inferred anomaly probabilities for the one or more input corpus data entries;
generating, using the keyword model, the explanatory metadata for the one or more inferred anomaly probabilities; and
initiating the performance of one or more prediction-based actions based at least in part on the one or more inferred anomaly probabilities and the explanatory metadata.
8. A system comprising memory and one or more processors communicatively coupled to the memory, the one or more processors configured to:
receive, via a plurality of exploration-exploitation keyword extraction iterations, a keyword list (i) based at least in part on a plurality of training corpus data entries and (ii) that satisfies a keyword list threshold condition, wherein an exploration-exploitation keyword extraction iteration of the plurality of exploration-exploitation keyword extraction iterations comprises:
(i) modifying the plurality of training corpus data entries by generating one or more anomalous data entries based at least in part on a prior per-iteration keyword list of a prior exploration-exploitation keyword extraction iteration that temporally precedes the exploration-exploitation keyword extraction iteration,
(ii) training a language-model-based binary classification model based at least in part on the one or more anomalous data entries, wherein the language-model-based binary classification model comprises:
(a) a bidirectional encoder layer that is configured to process the one or more anomalous data entries to generate an encoded representation,
(b) a dropout layer that is configured to process the encoded representation to generate a dropout representation of the input data object,
(c) a fully-connected layer that is configured to process the dropout representation to generate a fully connected output; and
(d) a softmax layer that is configured to process the fully connected output to generate an anomaly probability for the one or more anomalous data entries,
(iii) inputting the one or more anomalous data entries to the language-model-based binary classification model to generate one or more per-entry anomaly probabilities, and
(iv) generating a per-iteration keyword list for the exploration-exploitation keyword extraction iteration based at least in part on the one or more per-entry anomaly probabilities, wherein the keyword list comprises the per-iteration keyword list for an ultimate exploration-exploitation keyword extraction iteration of the plurality of exploration-exploitation keyword extraction iterations; and
update a keyword model associated with the language-model-based binary classification model based at least in part on the keyword list, wherein the keyword model is configured to provide explanatory metadata for one or more inferred anomaly probabilities output by the language-model-based binary classification model.
9. The system of claim 8 , wherein the prior exploration- exploitation keyword extraction iteration is an initial exploration-exploitation keyword extraction iteration of the plurality of exploration-exploitation keyword extraction iterations and the one or more anomalous data entries of the initial exploration-exploitation keyword extraction iteration are generated by removing randomly-selected removable words from one or more training corpus data entries of the plurality of training corpus data entries.
10. The system of claim 8 , wherein updating the keyword model comprises:
inputting a keyword of the keyword list to the bidirectional encoder layer to generate a per-keyword encoded representation of the keyword; and
generating, using a label-generating machine learning model, a labeled keyword cluster based at least in part on the per-keyword encoded representation.
11. The system of claim 10 , wherein generating the labeled keyword cluster comprises:
generating, using a principal component analysis layer, a reduced representation of the per-keyword encoded representation;
generating, using a t-distributed stochastic neighbor embedding layer, an unlabeled keyword cluster for the per-keyword encoded representation based at least in part on the reduced representation; and
generating, using a K-means layer, the labeled keyword cluster for the per-keyword encoded representation based on the unlabeled keyword cluster.
12. The system of claim 8 , wherein the keyword list threshold condition is satisfied when a keyword count for the per-iteration keyword list satisfies a threshold keyword count measure.
13. The system of claim 8 , wherein the keyword list threshold condition is satisfied when a deviation measure between the per-iteration keyword list and a prior per-iteration keyword list of the prior exploration-exploitation keyword extraction iteration satisfies a threshold deviation measure.
14. One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors, cause the one or more processors to:
receive, via a plurality of exploration-exploitation keyword extraction iterations, a keyword list (i) based at least in part on a plurality of training corpus data entries and (ii) that satisfies a keyword list threshold condition, wherein an exploration-exploitation keyword extraction iteration of the plurality of exploration-exploitation keyword extraction iterations comprises:
(i) modifying the plurality of training corpus data entries by generating one or more anomalous data entries based at least in part on a prior per-iteration keyword list of a prior exploration-exploitation keyword extraction iteration that temporally precedes the exploration-exploitation keyword extraction iteration,
(ii) training a language-model-based binary classification model based at least in part on the one or more anomalous data entries, wherein the language-model-based binary classification model comprises:
(a) a bidirectional encoder layer that is configured to process the one or more anomalous data entries to generate an encoded representation,
(b) a dropout layer that is configured to process the encoded representation to generate a dropout representation of the input data object,
(c) a fully-connected layer that is configured to process the dropout representation to generate a fully connected output; and
(d) a softmax layer that is configured to process the fully connected output to generate an anomaly probability for the one or more anomalous data entries,
(iii) inputting the one or more anomalous data entries to the language-model-based binary classification model to generate one or more per-entry anomaly probabilities, and
(iv) generating a per-iteration keyword list for the exploration-exploitation keyword extraction iteration based at least in part on the one or more per-entry anomaly probabilities, wherein the keyword list comprises the per-iteration keyword list for an ultimate exploration-exploitation keyword extraction iteration of the plurality of exploration-exploitation keyword extraction iterations; and
update a keyword model associated with the language-model-based binary classification model based at least in part on the keyword list, wherein the keyword model is configured to provide explanatory metadata for one or more inferred anomaly probabilities output by the language-model-based binary classification model.
15. The one or more non-transitory computer-readable storage media of claim 14 , wherein the prior exploration-exploitation keyword extraction iteration is an initial exploration-exploitation keyword extraction iteration of the plurality of exploration-exploitation keyword extraction iterations and the one or more anomalous data entries of the initial exploration-exploitation keyword extraction iteration are generated by removing randomly-selected removable words from one or more training corpus data entries of the plurality of training corpus data entries.