Generative artificial intelligence model prompt injection classifier
An analysis engine receives data characterizing a prompt for ingestion by a generative artificial intelligence (GenAI) model. The analysis engine, using a prompt injection classifier determines whether the prompt comprises or is indicative of malicious content or otherwise elicits malicious actions. Data characterizing the determination is provided to a consuming application or process. Related apparatus, systems, techniques and articles are also described.
1. A computer-implemented method comprising:
receiving, by an analysis engine from a model environment executing a generative artificial intelligence (GenAI) model, data characterizing a prompt for ingestion by the GenAI model, the analysis engine executing in a monitoring environment remote from the model environment;
determining, by the analysis engine using an ensemble of machine learning-based prompt injection classifiers, whether the prompt comprises malicious content or elicits malicious actions, a first of the prompt injection classifiers being trained to identify a first type of prompt injection attack and a second of the prompt injection classifiers being trained to identify a second, different type of prompt injection attack; and
providing data characterizing the determination to a consuming application or process to (i) initiate a remediation action to ensure that the GenAI model does not operate in an undesired manner when it is determined that the prompt comprises malicious content or elicits malicious actions and (ii) allow the prompt to be ingested when it is determined that the prompt does not comprise malicious content or elicit malicious actions;
wherein the initiated remediation action prevents the prompt from being ingested by the GenAI model to ensure that the GenAI model does not operate in an undesired manner and is based on whether the ensemble of machine learning-based prompt injection classifiers identifies the first type of prompt injection attack or the second type of prompt injection attack.
2. The method of claim 1 further comprising:
vectorizing the data characterizing the prompt to result in one or more vectors; and
generating one or more embeddings based on the one or more vectors, the embeddings having a lower dimensionality than the one or more vectors;
wherein at least one of the prompt injection classifiers uses the generated one or more embeddings when making the determination.
3. The method of claim 1 , wherein the GenAI model comprises a large language model.
4. The method of claim 1 , wherein the consuming application or process allows the prompt to be input into the GenAI model upon a determination that the prompt does not comprise or elicit malicious content.
5. The method of claim 1 , wherein the consuming application or process flags the prompt as being malicious for quality assurance upon a determination that the prompt comprises or elicits malicious content.
6. The method of claim 1 , wherein the consuming application or process modifies the prompt to be benign upon a determination that the prompt comprises or elicits malicious content and causes the modified prompt to be ingested by the GenAI model.
7. The method of claim 1 , wherein the consuming application or process blocks an internet protocol (IP) address of a requester of the prompt upon a determination that the prompt comprises or elicits malicious content.
8. The method of claim 1 , wherein the consuming application or process causes a subsequent prompt from an entity identified by one or more of an internet protocol (IP) address, a media access control (MAC) address, or a session identifier of a requester of the prompt to be modified upon a determination that the prompt comprises or elicits malicious content to result in a modified prompt and causes the modified prompt to be ingested by the GenAI model.
9. A system comprising:
at least one data processor; and
memory storing instructions which, when executed by the at least one data processor, result in operations comprising:
receiving, by an analysis engine from a model environment executing a generative artificial intelligence (GenAI) model, data characterizing a prompt for ingestion by the GenAI model, the analysis engine executing in a monitoring environment remote from the model environment;
determining, by the analysis engine using an ensemble of machine learning-based prompt injection classifiers, whether the prompt comprises malicious content or elicits malicious actions, a first of the prompt injection classifiers being trained to identify a first type of prompt injection attack and a second of the prompt injection classifiers being trained to identify a second, different type of prompt injection attack; and
providing data characterizing the determination to a consuming application or process to (i) initiate a remediation action to ensure that the GenAI model does not operate in an undesired manner when it is determined that the prompt comprises malicious content or elicits malicious actions and (ii) allow the prompt to be ingested when it is determined that the prompt does not comprise malicious content or elicit malicious actions;
wherein the initiated remediation action prevents the prompt from being ingested by the GenAI model to ensure that the GenAI model does not operate in an undesired manner and is based on whether the ensemble of machine learning-based prompt injection classifiers identifies the first type of prompt injection attack or the second type of prompt injection attack.
10. The system of claim 9 , wherein the operations further comprise:
vectorizing the data characterizing the prompt to result in one or more vectors; and
generating one or more embeddings based on the one or more vectors, the embeddings having a lower dimensionality than the one or more vectors;
wherein at least one of the prompt injection classifiers uses the generated one or more embeddings when making the determination.
11. The system of claim 9 , wherein the GenAI model comprises a large language model.
12. The system of claim 9 , wherein the consuming application or process allows the prompt to be input into the GenAI model upon a determination that the prompt does not comprise or elicit malicious content.
13. The system of claim 9 , wherein the consuming application or process flags the prompt as being malicious for quality assurance upon a determination that the prompt comprises or elicits malicious content.
14. The system of claim 9 , wherein the consuming application or process modifies the prompt to be benign upon a determination that the prompt comprises or elicits malicious content and causes the modified prompt to be ingested by the GenAI model.
15. The system of claim 9 , wherein the consuming application or process blocks an internet protocol (IP) address of a requester of the prompt upon a determination that the prompt comprises or elicits malicious content.
16. The system of claim 9 , wherein the consuming application or process causes a subsequent prompt from an entity identified by one or more of an internet protocol (IP) address, a media access control (MAC) address, or a session identifier of a requester of the prompt to be modified upon a determination that the prompt comprises or elicits malicious content to result in a modified prompt and causes the modified prompt to be ingested by the GenAI model.
17. A system comprising:
a model environment comprising memory and a plurality of data processors executing a generative artificial intelligence (GenAI) model; and
a monitoring environment comprising memory and a plurality of data processors executing an analysis engine, the monitoring environment being remote from the model environment;
wherein:
the analysis engine receives, from the model environment, data characterizing a prompt for ingestion by the GenAI model;
the analysis engine determines, using an ensemble of machine learning-based prompt injection classifiers, whether the prompt comprises malicious content or elicits malicious actions, a first of the prompt injection classifiers being trained to identify a first type of prompt injection attack and a second of the prompt injection classifiers being trained to identify a second, different type of prompt injection attack; and
data characterizing the determination is provided to a consuming application or process to (i) initiate a remediation action to ensure that the GenAI model does not operate in an undesired manner when it is determined that the prompt comprises malicious content or elicits malicious actions and (ii) allow the prompt to be ingested when it is determined that the prompt does not comprise malicious content or elicit malicious actions;
wherein the initiated remediation action prevents the prompt from being ingested by the GenAI model to ensure that the GenAI model does not operate in an undesired manner and is based on whether the ensemble of machine learning-based prompt injection classifiers identifies the first type of prompt injection attack or the second type of prompt injection attack.
18. The system of claim 17 , wherein the analysis engine:
vectorizes the data characterizing the prompt to result in one or more vectors; and
generates one or more embeddings based on the one or more vectors, the embeddings having a lower dimensionality than the one or more vectors;
wherein at least one of the prompt injection classifiers uses the generated one or more embeddings when making the determination.
19. The system of claim 17 , wherein the GenAI model comprises a large language model.
20. The system of claim 17 , wherein the consuming application or process allows the prompt to be input into the GenAI model upon a determination that the prompt does not comprise or elicit malicious content.
21. The system of claim 17 , wherein the consuming application or process flags the prompt as being malicious for quality assurance upon a determination that the prompt comprises or elicits malicious content.
22. The system of claim 17 , wherein the consuming application or process modifies the prompt to be benign upon a determination that the prompt comprises or elicits malicious content and causes the modified prompt to be ingested by the GenAI model.
23. The system of claim 17 , wherein the consuming application or process blocks an internet protocol (IP) address of a requester of the prompt upon a determination that the prompt comprises or elicits malicious content.
24. The system of claim 17 , wherein the consuming application or process causes a subsequent prompt from an entity identified by one or more of an internet protocol (IP) address, a media access control (MAC) address, or a session identifier of a requester of the prompt to be modified upon a determination that the prompt comprises or elicits malicious content to result in a modified prompt and causes the modified prompt to be ingested by the GenAI model.