Detecting jailbreak attempts on generative models
A computer-implemented method is provided that detects jailbreak attempts against generative models, which may involve a shift between benign and malicious content. The method includes determining a probability-based metric for each of a plurality of tokens in a target text using a language model, the probability-based metric being based on a probability at least one preceding token. The probability-based metrics are processed to identify a subset of the plurality of tokens having a change in the probability-based metric with respect to others of the plurality of the tokens not within the subset of the plurality of tokens. A jailbreak attempt in the target text is detected in response to identifying the change in the probability-based metric in the subset of the plurality of tokens.
1 . A computer-implemented method of using a first generative model to detect a jailbreak attempt in target data for input to or being output from a second generative model, the method comprising:
determining, for a token in a plurality of tokens of the target data, a probability-based metric using the first generative model, the probability-based metric of the token of the plurality of tokens comprising a probability combined with a metric, the probability being a probability of the token being generated by the second generative model, the probability-based metric accounting for a probability of the token being generated by the second generative model based on a preceding token in the plurality of tokens;
identifying a portion of the plurality of tokens having a change in the probability-based metric with respect to another of the plurality of the tokens not within the portion of the plurality of tokens, the change being reflective of a reduced probability of the tokens being generated by the second generative model in the portion of the plurality of tokens; and
detecting the jailbreak attempt in the target data in response to identifying the change in the probability-based metric in the portion of the plurality of tokens based on the reduced probability indicating a shift from a benign content of the target data to a malicious content of the target data.
2 . The method of claim 1 , wherein the target data is a text.
3 . The method of claim 1 , wherein the target data is a response output by the second generative model.
4 . The method of claim 1 , comprising performing a security action in response to detecting the jailbreak attempt.
5 . The method of claim 4 , wherein the security action is one of: preventing display of the target data; preventing an action carried out based on a response of the second generative model; generating an alert; generating a log entry; and changing user rights.
6 . The method of claim 1 , wherein the detected jailbreak attempt in the target data comprises an indirect prompt injection.
7 . The method of claim 1 , wherein the at least one preceding token is every preceding token of the plurality of tokens.
8 . The method of claim 1 , further comprising the first generative model and the second generative model being a same model and the first generative model generating the probability-based metric to detect the jailbreak attempt on the first generative model.
9 . The method of claim 1 , comprising:
the portion of the plurality of tokens comprising at least two tokens;
processing a plurality of portions of the plurality of tokens; and
detecting the jailbreak attempt in response to identifying a change in the probability-based metric in any of the portions.
10 . The method of claim 9 , further comprising:
the probability-based metric being a perplexity calculated for the token based on using each previous token in the plurality of tokens, including the preceding token, to identify a shift in perplexity, the perplexity being calculated as a function of T, x, and P LM , T being a number of the plurality of tokens, x representing each of the tokens in the plurality of tokens, and PLM being the probability of each token being generated by the second generative model given by the first generative model;
the portion of the plurality of tokens comprising a window having a window size of a predetermined number of tokens; and
identifying the portion of the plurality of tokens having the change in probability-based metric comprising sliding the window over the plurality of tokens.
11 . The method of claim 1 , wherein the first generative model is a multimodal model configured to receive the target data in a first modality provide an output in a second modality, the first modality and the second modality being different.
12 . The method of claim 1 , wherein the target data is a first target data and the plurality of tokens are a first plurality of tokens, the first target data being a prompt for input to the second generative model;
wherein the method comprises:
providing the first target data as input to the second generative model;
receiving from the second generative model:
a response comprising a second plurality of tokens; and
probability values associated with each of the first plurality of tokens and second plurality of tokens; and
determining the probability-based metrics of the first plurality of tokens and the second plurality of tokens based on the probability values.
13 . The method of claim 1 , wherein identifying the change in the probability-based metric in the portion of the plurality of tokens comprises:
determining that a first token of the portion has a probability-based metric reflective of a probability below a first threshold; and
determining that a second token of the portion has a probability-based metric reflective of a probability below a second threshold.
14 . The method of claim 13 , wherein the first token forms a beginning of the portion, and the second token forms an end of the portion.
15 . The method of claim 13 , comprising determining the second threshold based on the first threshold.
16 . A system comprising a processor and a memory, the memory storing computer-readable instructions, which when executed by the processor, cause the system to carry out operations comprising:
determining, for a token in a plurality of tokens of target data, a probability-based metric using a first generative model, the probability-based metric of the token of the plurality of tokens comprising a probability combined with a metric, the probability being a probability of the token being generated by a second generative model, the probability-based metric accounting for a probability of the token being generated by the second generative model based on a preceding token in the plurality of tokens;
identifying a portion of the plurality of tokens having a change in the probability-based metric with respect to another of the plurality of the tokens not within the portion of the plurality of tokens, the change being reflective of a reduced probability of the tokens being generated by the second generative model in the portion of the plurality of tokens; and
detecting a jailbreak attempt in the target data in response to identifying the change in the probability-based metric in the portion of the plurality of tokens based on the reduced probability indicating a shift from a benign content of the target data to a malicious content of the target data.
17 . The system of claim 16 , wherein the target data is a response output by the second generative model.
18 . The system of claim 16 , wherein the target data is a text.
19 . The system of claim 16 , wherein the target data is a sequence of images.
20 . A non-transitory computer-readable medium storing instructions, which when executed by a processor, cause the processor to carry out operations comprising:
determining, for a token in a plurality of tokens of target data, a probability-based metric using a first generative model, the probability-based metric of the token of the plurality of tokens comprising a probability combined with a metric, the probability being a probability of the token being generated by a second generative model, the probability-based metric accounting for a probability of the token being generated by the second generative model based on a preceding token in the plurality of tokens;
identifying a portion of the plurality of tokens having a change in the probability-based metric with respect to another of the plurality of the tokens not within the portion of the plurality of tokens, the change being reflective of a reduced probability of the tokens being generated by the second generative model in the portion of the plurality of tokens;
detecting a jailbreak attempt in the target data in response to identifying the change in the probability-based metric in the portion of the plurality of tokens based on the reduced probability indicating a shift from a benign content of the target data to a malicious content of the target data; and
further comprising the first generative model and the second generative model being a same model, and the first generative model generating the probability-based metric to detect the jailbreak attempt on the first generative model.