System and method for detecting prompt injection attacks to large language models

1 . A computer-implemented method, comprising:

receiving, a large language model system comprising a large language model and a trained neural network, a prompt to be provided as input to the large language model;

generating, by the large language model system, an input string by appending a plurality of contexts to the prompt, the plurality of contexts defining rules for the large language model to follow when processing the prompt, the plurality of contexts comprising a negative context and a positive context;

based on the prompt and the plurality of contexts, determining, by the large language model system, at least one relationship between the prompt, the positive context, and the negative context;

generating, by the large language model system, an attention matrix representing the at least one relationship between the prompt and the plurality of contexts;

providing, by the large language model system, the attention matrix to the trained neural network to determine a likelihood that the prompt is malicious, wherein the trained neural network is separate from the large language model; and

responsive to determining that the prompt is likely a malicious prompt, initiating, by the large language model, a remedial action, wherein the remedial action includes at least preventing the prompt from being transmitted to the large language model.

2 . The computer-implemented method of claim 1 , wherein generating the attention matrix comprises:

grouping the plurality of contexts into at least one of a plurality of positive contexts or a plurality of negative contexts.

3 . The computer-implemented method of claim 2 , further comprising:

determining that the prompt is likely a malicious prompt based on a first correlation between the prompt and the plurality of negative contexts and a second correlation between the prompt and the plurality of positive contexts.

4 . The computer-implemented method of claim 1 , further comprising:

for each cell in the attention matrix, generating a sub-matrix comparing at least one of the prompt or the plurality of contexts to at least one of the prompt or the plurality of contexts.

5 . The computer-implemented method of claim 4 , further comprising:

generating an attention score for each sub-matrix by aggregating over the sub-matrix.

6 . The computer-implemented method of claim 1 , further comprising:

individually tokenizing each of the prompt and the plurality of contexts prior to appending the plurality of contexts to the prompt.

7 . A computer-implemented method, comprising:

generating a training data set, by a large language model system comprising a large language model and a neural network, for training the neural network to detect a prompt injection attack to the large language model, the generating comprising:

obtaining, by the large language model system, a plurality of contexts, the plurality of contexts defining rules for the large language model to follow when processing prompts, the plurality of contexts comprising a plurality of negative contexts and a plurality of positive contexts;

obtaining, by the large language model system, a plurality of training prompts for each context, the plurality of training prompts comprising a first set of non-malicious training prompts associated with the context and a second set of malicious training prompts associated with the context;

for each training prompt of the plurality of training prompts,

generating, by the large language model system, input strings to the large language model by appending the plurality of contexts to the training prompt,

determining at least one relationship between the plurality of contexts and the training prompt;

generating, by the large language model system, attention matrices based on the input strings, the attention matrices indicating the at least one relationships between the training prompt and the plurality of contexts, and

tagging, by the large language model system, each attention matrix with an indication whether the training prompt associated with the attention matrix is malicious or non-malicious, wherein the attention matrices and corresponding tags form the training data set; and

based on the training data set, training the neural network to detect the prompt injection attack.

8 . The computer-implemented method of claim 7 , further comprising:

deploying the neural network in a content management system comprising the large language model system.

9 . The computer-implemented method of claim 7 , wherein the neural network is trained to detect the prompt injection attack by learning correlations between prompts and negative contexts.

10 . The computer-implemented method of claim 7 , further comprising:

for each cell in the attention matrix, generating a sub-matrix comparing at least one of the training prompt or the plurality of contexts to at least one of the training prompt or the plurality of contexts.

11 . The computer-implemented method of claim 10 , further comprising:

generating an attention score for each sub-matrix by aggregating over the sub-matrix.

12 . The computer-implemented method of claim 7 , further comprising:

individually tokenizing each of the training prompt and the plurality of contexts prior to appending the plurality of contexts to the training prompt.

13 . The computer-implemented method of claim 7 , the training data set comprises a second plurality of contexts, the second plurality of contexts defining universal rules associated with the large language model.

14 . A computer-implemented method, comprising:

intercepting a prompt destined as input to a large language model;

generating, by a large language model system, an input string to a machine learning model by appending a plurality of contexts to the prompt, the plurality of contexts defining rules to be enforced by the machine learning model, the rules defining whether the prompt can reach the large language model, the plurality of contexts comprising a negative context and a positive context;

based on the prompt and the plurality of contexts, determining, by the large language model, a relationship data set based on the prompt, the positive context, and the negative context

generating, by the large language model, the relationship data set representing relationships between the prompt and the plurality of contexts;

providing, by the large language model system, the relationship data set to a trained neural network to determine a likelihood that the prompt is malicious, wherein the trained neural network is separate from the large language model; and

responsive to determining that the prompt is likely a malicious prompt, preventing, by the large language model system, transmission of the prompt to the large language model.

15 . The computer-implemented method of claim 14 , wherein generating the relationship data set comprises:

grouping the plurality of contexts into at least one of a plurality of positive contexts or a plurality of negative contexts.

16 . The computer-implemented method of claim 15 , further comprising:

determining that the prompt is likely the malicious prompt based on a first correlation between the prompt and the plurality of negative contexts and a second correlation between the prompt and the plurality of positive contexts.

17 . The computer-implemented method of claim 14 , wherein the relationship data set is an attention matrix, and the computer-implemented method further comprises:

for each cell in the attention matrix, generating a sub-matrix comparing at least one of the prompt or the plurality of contexts to at least one of the prompt or the plurality of contexts.

18 . The computer-implemented method of claim 17 , further comprising:

generating an attention score for each sub-matrix by aggregating over the sub-matrix.