Detecting evasive prompts for generative artificial intelligence systems
A genetic algorithm is implemented to generate prompts that evade content filters of generative artificial intelligence (AI) systems. The genetic algorithm applies grammar operations to mutate candidate prompts, communicates the candidate prompts to generative AI systems, and selects candidate prompts that successfully evade content filters according to corresponding responses. A disambiguation model that corrects grammar in prompts is tested on the selected prompts to determine if grammar is properly corrected. Once tested, the disambiguation model is deployed in an ensemble with a classifier that outputs verdicts for prompts with grammar corrected by the disambiguation model.
1 . A method comprising: generating evasive prompts that evade content filters for one or more generative artificial intelligence (AI) systems with a genetic algorithm, wherein generating the evasive prompts with the genetic algorithm comprises, for each iteration of the genetic algorithm and corresponding set of candidate prompts and set of prompt genealogies at the iteration, selecting a mutation operation at random for each prompt genealogy of the set of prompt genealogies; and
applying the selected mutation operations to the set of candidate prompts to generate additional candidate prompts for a subsequent iteration; testing a first foundation model on disambiguating the evasive prompts; and based on determining that the first foundation model successfully disambiguated the evasive prompts, intercepting a first prompt destined for a second foundation model;
prompting the first foundation model to disambiguate the first prompt to generate a disambiguated first prompt; inputting at least one of the disambiguated first prompt and a summary of the first prompt into a classifier to obtain a verdict as output; and based on obtaining the verdict for the disambiguated first prompt that indicates that the first prompt is at least one of malicious and evasive, blocking the first prompt from being communicated to the second foundation model.
2 . The method of claim 1 , wherein the mutation operation for the genetic algorithm comprises a grammar operation.
3 . The method of claim 2 , wherein the grammar operation comprises at least one of prepending a phrase, appending a phrase, adding a line feed, repeating a keyword, adding a relative word, and removing a word.
4 . The method of claim 1 , further comprising testing the evasive prompts generated with the genetic algorithm, wherein testing the evasive prompts comprises:
communicating the evasive prompts to one or more of the one or more generative AI systems; and
evaluating responses to the evasive prompts to determine at least one of whether the evasive prompts were blocked and whether the responses are responsive to content in the evasive prompts.
5 . The method of claim 4 , wherein a fitness function for the genetic algorithm comprises a metric of negative tone in the responses to the evasive prompts.
6 . The method of claim 1 , wherein the first foundation model comprises a large language model.
7 . A non-transitory machine-readable medium having program code stored thereon, the program code comprising instructions to: generate evasive prompts that evade content filters for one or more generative artificial intelligence (AI) systems with a genetic algorithm, wherein the instructions to generate the evasive prompts with the genetic algorithm comprise instructions to, for each iteration of the genetic algorithm and corresponding set of candidate prompts and set of prompt genealogies at the iteration, select a mutation operation at random for each prompt genealogy of the set of prompt genealogies; and apply the selected mutation operations to the set of candidate prompts to generate additional candidate prompts for a subsequent iteration; test a first foundation model on disambiguating the evasive prompts; and based on a determination that the first foundation model successfully disambiguated the evasive prompts, intercept a first prompt destined for a second foundation model; prompt the first foundation model to disambiguate the first prompt; obtain a verdict for the first prompt based on output of the first foundation model from the prompting; and block or allow the first prompt based on the verdict.
8 . The non-transitory machine-readable medium of claim 7 , wherein the mutation operation for the genetic algorithm comprises a grammar operation.
9 . The non-transitory machine-readable medium of claim 8 , wherein the grammar operation comprises at least one of prepending a phrase, appending a phrase, adding a line feed, repeating a keyword, adding a relative word, and removing a word.
10 . The non-transitory machine-readable medium of claim 7 , wherein the program code further comprises instructions to test prompts generated with the genetic algorithm, wherein the program code to test the prompts comprises instructions to:
communicate the prompts to one or more of the generative AI systems; and
evaluate responses to the prompts to determine at least one of whether the prompts were blocked and whether the responses are responsive to content in the prompts.
11 . An apparatus comprising: a processor; and a machine-readable medium having instructions stored thereon that are executable by the processor to cause the apparatus to, maintain a database of prompts known to evade content filters of generative artificial intelligence (AI) systems, wherein the instructions to maintain the database of prompts known to evade generative AI systems comprise instructions executable by the processor to cause the apparatus to generate the prompts with a genetic algorithm, wherein the instructions to generate the prompts known to evade the content filters of generative AI systems with the genetic algorithm comprise instructions executable by the processor to cause the apparatus to, for each iteration of the genetic algorithm and corresponding set of candidate prompts and set of prompt genealogies at the iteration, select a mutation operation at random for each prompt genealogy of the set of prompt genealogies; and apply the selected mutation operations to the set of candidate prompts to generate additional candidate prompts for a subsequent iteration; test a first foundation model on prompts in the database to determine whether the first foundation model can correct grammar in the prompts; and based on a determination that the first foundation model can correct grammar in the prompts, deploy the first foundation model in an ensemble with a classifier to intercept and detect evasive prompts intended for a second foundation model, wherein the instructions to detect the evasive prompts comprise instructions executable by the processor to cause the apparatus to: prompt the first foundation model with the evasive prompts to obtain corrected versions of the evasive prompts; and input the corrected versions of the evasive prompts into the classifier to obtain evasive verdicts.
12 . The apparatus of claim 11 , wherein the mutation operation for the genetic algorithm comprises a grammar operation.
13 . The apparatus of claim 12 , wherein the grammar operation comprises at least one of prepending a phrase, appending a phrase, adding a line feed, repeating a keyword, adding a relative word, and removing a word.
14 . The apparatus of claim 11 , wherein the machine-readable medium further has stored thereon instructions executable by the processor to cause the apparatus to test prompts generated with the genetic algorithm, wherein the instructions to test the prompts comprise instructions executable by the processor to cause the apparatus to:
communicate the prompts to one or more of the generative AI systems; and
evaluate responses to the prompts to determine at least one of whether the prompts were blocked and whether the responses are responsive to content in the prompts.
15 . The method of claim 1 , wherein prompting the first foundation model to disambiguate the first prompt comprises prompting the first foundation model with a second prompt comprising at least one of an instruction to correct grammar in the first prompt and an instruction to generate a summary of the first prompt with corrected grammar.
16 . The method of claim 1 , wherein generating the evasive prompts with the genetic algorithm further comprises, for each iteration of the genetic algorithm, performing one or more crossover operations on those of the additional candidate prompts that evade the content filters of the one or more generative AI systems to seed the subsequent iteration of the genetic algorithm.
17 . The non-transitory machine-readable medium of claim 7 , wherein the instructions to prompt the first foundation model to disambiguate the first prompt comprise instructions to prompt the first foundation model with a second prompt comprising task instructions to at least one of correct grammar in the first prompt, generate a summary of the first prompt with corrected grammar, and identify a number of grammar errors in the first prompt, wherein the instructions to obtain the verdict for the first prompt comprise instructions to:
determine whether a number of grammar errors in the first prompt indicated by the first foundation model is above a threshold;
based on the number of grammar errors being above the threshold, and indicate at least one of an evasive verdict and a malicious verdict if the threshold is satisfied; and
based on the number of grammar errors being below the threshold, invoke a classifier on outputs of the second foundation model if the threshold is not satisfied.
18 . The non-transitory machine-readable medium of claim 7 , wherein the instructions to generate the evasive prompts with the genetic algorithm further comprise instructions to, for each iteration of the genetic algorithm, perform one or more crossover operations on those of the additional candidate prompts that evade the content filters of the one or more generative AI systems to seed the subsequent iteration of the genetic algorithm.
19 . The apparatus of claim 11 , wherein the instructions to generate the prompts known to evade the content filters of generative AI systems with the genetic algorithm further comprise instructions executable by the processor to cause the apparatus to, for each iteration of the genetic algorithm, perform one or more crossover operations on those of the additional candidate prompts that evade the content filters of generative AI systems to seed the subsequent iteration of the genetic algorithm.
20 . The apparatus of claim 14 , wherein a fitness function for the genetic algorithm comprises a metric of negative tone in the responses to the prompts.