Large language model prompt security with clustering
A clustering model clusters multiple LLM prompts through a clustering model to generate multiple clusters. Multiple judge models classify the LLM prompts to obtain multiple predictions for each LLM prompt of the LLM prompts. The predictions are into a first set of security classes corresponding to at least one malicious class and a second set of security classes corresponding to at least one benign class. Multiple sample LLM prompts in a cluster are selected based on a discrepancy amongst the predictions of each sample LLM prompt in the sample LLM prompts. A lead model evaluates the sample LLM prompts to select a judge model of the judge models for the cluster. The selected judge model is assigned to the cluster.
1 . A method comprising:
clustering, by a clustering model, a plurality of large language model (LLM) prompts through the clustering model to generate a plurality of clusters;
classifying, by a plurality of judge models, the plurality of LLM prompts to obtain a plurality of predictions for each LLM prompt of the plurality of LLM prompts, wherein the plurality of predictions is into a first set of security classes corresponding to at least one malicious class and a second set of security classes corresponding to at least one benign class, wherein the plurality of predictions are separate from the clustering of the plurality of LLM prompts;
selecting a plurality of sample LLM prompts in a cluster of the plurality of clusters based on a discrepancy amongst the plurality of predictions of each sample LLM prompt in the plurality of sample LLM prompts;
evaluating, by a lead model, the plurality of sample LLM prompts to select a judge model of the plurality of judge models for the cluster based on consistency in prediction with the lead model; and
assigning the selected judge model to the cluster, wherein the selected judge model performs a prediction for new LLM prompts assigned to the cluster to cause a routing of the new LLM prompts.
2 . The method of claim 1 , further comprising:
selecting the cluster of the plurality of clusters using a new LLM prompt;
identifying the selected judge model assigned to the cluster;
processing the new LLM prompt through the selected judge model to obtain a judge selected security class from the first set of security classes and the second set of security classes; and
routing the new LLM prompt according to the judge selected security class.
3 . The method of claim 2 , further comprising:
processing the new LLM prompt through a first stage model to generate a prediction for the new LLM prompt; and
determining that the prediction fails to satisfy a class threshold for a particular security class of a plurality of classes,
wherein the processing the new LLM prompt through the selected judge model is responsive to the prediction failing to satisfy the class threshold.
4 . The method of claim 2 , wherein selecting the cluster comprises:
extracting a plurality of features of the new LLM prompt; and
selecting the cluster according to the plurality of features.
5 . The method of claim 1 , further comprising:
training a first stage model with a first stage labeled training dataset to generate a trained first stage model; and
processing, by the trained first stage model, a second stage unlabeled training dataset to filter the second stage unlabeled training dataset,
wherein the second stage unlabeled training dataset comprises the plurality of LLM prompts after filtering.
6 . The method of claim 1 , wherein the plurality of sample LLM prompts is a predefined portion of the plurality of predictions in the cluster having the discrepancy satisfying a discrepancy threshold.
7 . The method of claim 1 , further comprising:
determining a consensus rate for each cluster of the plurality of clusters using the plurality of predictions for each LLM prompt; and
assigning a plurality of clusters satisfying the consensus rate to at least one judge model based on resource usage of the at least one judge model.
8 . A system comprising:
at least one computer processor;
a clustering model executing on the at least one computer processor for causing the at least one computer processor to:
cluster a plurality of large language model (LLM) prompts to generate a plurality of clusters;
a plurality of judge models executing on the at least one computer processor for causing the at least one computer processor to:
classify the plurality of LLM prompts to obtain a plurality of predictions for each LLM prompt of the plurality of LLM prompts, wherein the plurality of predictions is into a first set of security classes corresponding to at least one malicious class and a second set of security classes corresponding to at least one benign class, wherein the plurality of predictions are separate from the clustering of the plurality of LLM prompts;
a discrepancy detector executing on the computer processor and configured to select a plurality of sample LLM prompts in a cluster of the plurality of clusters based on a discrepancy amongst the plurality of predictions of each sample LLM prompt in the plurality of sample LLM prompts;
a lead model executing on the at least one computer processor and configured to:
evaluate the plurality of sample LLM prompts to select a judge model of the plurality of judge models for the cluster based on consistency in prediction with the lead model; and
a training program executing on the at least one computer processor configured to assign the selected judge model to the cluster, wherein the selected judge model performs a prediction for new LLM prompts assigned to the cluster to cause a routing of the new LLM prompts.
9 . The system of claim 8 , further comprising:
a cluster assignment process executing on the at least one computer processor and configured to:
select the cluster of the plurality of clusters using a new LLM prompt, and
identify the selected judge model assigned to the cluster;
wherein the selected judge model is further configured to process the new LLM prompt through the selected judge model to obtain a judge selected security class from the first set of security classes and the second set of security classes; and
a network routing program configured to route the new LLM prompt according to the judge selected security class.
10 . The system of claim 9 , further comprising:
a first stage router executing on the at least one computer processor and configured to:
process the new LLM prompt through a first stage model to generate a prediction for the new LLM prompt; and
determine that the prediction fails to satisfy a class threshold for a particular security class of a plurality of classes,
wherein the selected judge model processes the new LLM prompt responsive to the prediction failing to satisfy the class threshold.
11 . The system of claim 9 , wherein selecting the cluster comprises:
extracting a plurality of features of the new LLM prompt; and
selecting the cluster according to the plurality of features.
12 . The system of claim 8 , further comprising:
a first stage training program operating with a first stage router executing on the at least one computer processor and configured to:
train a first stage model with a first stage labeled training dataset to generate a trained first stage model, and
process, by the trained first stage model, a second stage unlabeled training dataset to filter the second stage unlabeled training dataset,
wherein the second stage unlabeled training dataset comprises the plurality of LLM prompts after filtering.
13 . The system of claim 8 , wherein the plurality of sample LLM prompts is a predefined portion of the plurality of predictions in the cluster having the discrepancy satisfying a discrepancy threshold.
14 . The system of claim 8 , further comprising:
a second stage training program configured to:
determining a consensus rate for each cluster of the plurality of clusters using the plurality of predictions for each LLM prompt, and
assigning a plurality of clusters satisfying the consensus rate to at least one judge model based on resource usage of the at least one judge model.
15 . A method comprising:
clustering, by a clustering model, a plurality of user large language model (LLM) prompts through the clustering model to generate a plurality of clusters;
classifying, by a plurality of judge models, the plurality of user LLM prompts to obtain a plurality of predictions for each user LLM prompt of the plurality of user LLM prompts, wherein the plurality of predictions is into a first security class comprising a prompt injection attack class and a second security class comprising a benign class, wherein the plurality of predictions are separate from the clustering of the plurality of user LLM prompts;
selecting a plurality of sample user LLM prompts in a cluster of the plurality of clusters based on a discrepancy amongst the plurality of predictions of each sample user LLM prompt in the plurality of sample user LLM prompts;
evaluating, by a lead model, the plurality of sample user LLM prompts to select a judge model of the plurality of judge models for the cluster based on consistency in prediction with the lead model; and
assigning the selected judge model to the cluster, wherein the selected judge model performs a prediction for new LLM prompts assigned to the cluster to cause a routing of the new LLM prompts.
16 . The method of claim 15 , further comprising:
selecting the cluster of the plurality of clusters using a new user LLM prompt;
identifying the selected judge model assigned to the cluster;
processing the new user LLM prompt through the selected judge model to obtain a judge selected security class from the first security class and the second security class; and
routing the new user LLM prompt according to the judge selected security class, wherein routing the new user LLM prompt comprises:
logging the new user LLM prompt, and
dropping the new user LLM prompt without transmitting the new user LLM prompt to an LLM responsive to the judge selected security class being the prompt injection attack class.
17 . The method of claim 16 , further comprising:
processing the new user LLM prompt through a first stage model to generate a prediction for the new user LLM prompt; and
determining that the prediction fails to satisfy a class threshold for the benign class,
wherein the processing the new user LLM prompt through the selected judge model is responsive to the prediction failing to satisfy the class threshold.
18 . The method of claim 16 , wherein selecting the cluster comprises:
extracting a plurality of features of the new user LLM prompt; and
selecting the cluster according to the plurality of features.
19 . The method of claim 15 , further comprising:
training a first stage model with a first stage labeled training dataset to generate a trained first stage model; and
processing, by the trained first stage model, a second stage unlabeled training dataset to filter the second stage unlabeled training dataset,
wherein the second stage unlabeled training dataset comprises the plurality of user LLM prompts after filtering.
20 . The method of claim 15 , further comprising:
determining a consensus rate for each cluster of the plurality of clusters using the plurality of predictions for each user LLM prompt; and
assigning a plurality of clusters satisfying the consensus rate to at least one judge model based on resource usage of the at least one judge model.