GenAI prompt injection classifier training using prompt attack structures
An analysis engine receives data characterizing a prompt for ingestion by a generative artificial intelligence (GenAI) model. The analysis engine, using a prompt injection classifier determines whether the prompt comprises or is indicative of malicious content or otherwise elicits malicious actions. The prompt injection classifier can be trained using a dataset generated by populating benign content and malicious content into a plurality of different prompt attack structures at pre-defined locations. Data characterizing the determination is provided to a consuming application or process. Related apparatus, systems, techniques and articles are also described.
1 . A computer-implemented method comprising:
receiving data characterizing a prompt for ingestion by a generative artificial intelligence (GenAI) model;
determining, using a prompt injection classifier, whether the prompt comprises malicious content or elicits malicious actions; and
providing data characterizing the determination to a consuming application or process;
wherein the prompt injection classifier is trained using a dataset generated by at least one large language model populating benign content and malicious content into a plurality of different prompt attack structures.
2 . The method of claim 1 , wherein at least a portion of the malicious content is generated by instructing a misaligned large language model to generate a plurality of strings having malicious content or eliciting malicious actions, the misaligned large language model having been fine-tuned to output malicious strings.
3 . The method of claim 1 , wherein at least a portion of the malicious content is generated by instructing a jailbroken large language model to generate a plurality of strings having malicious content or eliciting malicious actions, the jailbroken large language model having been prompted with a specific input that allows it to respond with malicious strings.
4 . The method of claim 1 , wherein at least a portion of the malicious content is generated by instructing a large language model to generate a plurality of strings having malicious content or eliciting malicious actions, the large language model having not been aligned in a way that restricts its output.
5 . The method of claim 1 , wherein at least a portion of the benign content is generated using a large language model with guardrails to generate a plurality of strings known to be benign.
6 . The method of claim 1 , wherein at least a portion of the benign content comprises human-generated content.
7 . The method of claim 1 , wherein at least a portion of the malicious content comprises human-generated content.
8 . The method of claim 1 , wherein the GenAI model comprises a large language model.
9 . The method of claim 1 , wherein the consuming application or process prevents the prompt from being input into the GenAI model upon a determination that the prompt comprises malicious content or elicits malicious actions content.
10 . The method of claim 1 , wherein the consuming application or process allows the prompt to be input into the GenAI model upon a determination that the prompt does not comprise malicious content or elicit malicious actions content.
11 . The method of claim 1 , wherein the consuming application or process flags the prompt as being malicious for quality assurance upon a determination that the prompt comprises malicious content or elicits malicious actions content.
12 . The method of claim 1 , wherein the consuming application or process modifies the prompt to be benign upon a determination that the prompt comprises malicious content or elicits malicious actions content and causes the modified prompt to be ingested by the GenAI model.
13 . The method of claim 1 , wherein the consuming application or process blocks an internet protocol (IP) address of a requester of the prompt upon a determination that the prompt comprises malicious content or elicits malicious actions content.
14 . The method of claim 1 , wherein the consuming application or process causes subsequent prompts from an entity identified by one or more of an internet protocol (IP) address, a media access control (MAC) address, or a session identifier of a requester of the prompt to be modified upon a determination that the prompt comprises malicious content or elicits malicious actions content and causes the modified prompt to be ingested by the GenAI model.
15 . The method of claim 1 , wherein the data characterizing the prompt is vectorized to generate sentence embeddings, and the prompt injection classifier determines whether the prompt comprises malicious content or elicits malicious actions based at least in part on features derived from the embeddings.
16 . The method of claim 1 , wherein the prompt injection classifier outputs a confidence score indicative of whether the prompt comprises malicious content or elicits malicious actions, and providing data characterizing the determination comprises providing the confidence score to the consuming application or process.
17 . The method of claim 1 , wherein determining uses an ensemble comprising a binary prompt injection classifier executed prior to a multi-class prompt injection classifier that, responsive to the binary classifier indicating the prompt is malicious, identifies a type of prompt injection attack for the prompt.
18 . The method of claim 17 , wherein the type of prompt injection attack is selected from a direct task deflection attack, a special case attack, a context continuation attack, a context termination attack, a syntactic transformation attack, an encryption attack, or a text redirection attack.
19 . The method of claim 1 , wherein the plurality of different prompt attack structures comprises templates selected from defined dictionary templates, context switching templates, and jailbreak templates.
20 . The method of claim 1 , wherein the prompt injection classifier is executed by an analysis engine in a monitoring environment communicatively coupled to a proxy interposed between client devices and a model environment hosting the GenAI model, the proxy relaying prompts or features characterizing prompts for analysis prior to ingestion by the GenAI model.
21 . The method of claim 20 , wherein the proxy intercepts prompts from the client devices and relays the prompts or features characterizing the prompts to the analysis engine in the monitoring environment prior to the prompts being input into the GenAI model.
22 . The method of claim 20 , wherein the proxy further relays outputs of the GenAI model to the monitoring environment and the determining is based on at least one of the prompt and the output.
23 . The method of claim 1 , wherein providing data characterizing the determination to the consuming application or process causes generation of an alert and capture of system or process behavior associated with the prompt.
24 . The method of claim 1 , wherein the prompt injection classifier is executed by a local analysis engine within a model environment and the determination is transmitted to a monitoring environment or external remediation resources for remediation.
25 . A computer-implemented method comprising:
training a prompt injection classifier using a dataset generated by at least one large language model to populate benign content and malicious content into a plurality of different prompt attack structures at pre-defined locations; and
causing the trained prompt injection classifier to be deployed to determine whether prompts to be ingested by a generative artificial intelligence (GenAI) model comprise malicious content or elicits malicious actions.
26 . The method of claim 25 , wherein at least a portion of the malicious content is generated by instructing a misaligned large language model to generate a plurality of strings having malicious content or eliciting malicious actions, the misaligned large language model having been fine-tuned to output malicious strings.
27 . The method of claim 25 , wherein at least a portion of the malicious content is generated by instructing a jailbroken large language model to generate a plurality of strings having malicious content or eliciting malicious actions, the jailbroken large language model having been prompted with a specific input that allows it to respond with malicious strings.
28 . The method of claim 25 , wherein at least a portion of the malicious content is generated by instructing a large language model to generate a plurality of strings having malicious content or eliciting malicious actions, the large language model having not been aligned in a way that restricts its output.
29 . A computer-implemented method comprising:
generating malicious content by instructing a misaligned or jailbroken first large language model to generate malicious strings having malicious content or eliciting malicious actions;
generating benign content by instructing a second large language model to generate benign strings having benign content; and
generating at least a portion of a training dataset by using a large language model to populate a plurality of different prompt attack structures with generated benign strings at locations tagged in the structures as being benign and with generated malicious strings at locations tagged in the structures as being malicious.
30 . The method of claim 29 further comprising:
training a prompt injection classifier using the training dataset; and
deploying the trained prompt injection classifier to determine whether prompts to be ingested by a generative artificial intelligence (GenAI) model comprise malicious content or elicits malicious actions.