Detecting and mitigating injection of indirect instructions for trained models
Disclosed embodiments relate to systems and methods for detecting and mitigating unwanted instructions in artificial intelligence models authentication at a client application. Example techniques include receiving, an original request from a user; identifying historical data associated with a previous interaction with the user; receiving, from a trained model, an action identifier associated with raw data stored in a data source; receiving the raw data from the data source as a result of performing an action based on the action identifier; supplementing the raw data with a relevant portion of the historical data to generate supplemented data; determining whether the supplemented data includes an instruction; and based on a determination that the supplemented data includes the instruction, performing a control action.
1 . A non-transitory computer readable medium including instructions that, when executed by at least one processor, cause the at least one processor to perform operations for detecting and mitigating indirect instructions provided to a trained model, the operations comprising:
receiving, at an orchestrator application associated with a first trained model, an original request from a user;
identifying historical data associated with one or more previous interactions with the user;
receiving, from the first trained model, at least one action identifier associated with raw data stored in at least one data source, the action identifier specifying at least one action to be performed using the at least one data source;
receiving the raw data from the at least one data source as a result of performing the at least one action based on the at least one action identifier;
supplementing the raw data with a relevant portion of the historical data to generate supplemented data;
determining, based on the supplemented data, whether the supplemented data includes an instruction;
based on a determination that the supplemented data does not include the instruction, storing formatted data, the original request, and the raw data in a repository, the formatted data being based on the raw data, the original request, and the historical data; and
based on a determination that the supplemented data includes the instruction, performing a control action.
2 . The non-transitory computer readable medium of claim 1 , wherein the at least one data source is inaccessible to the first trained model.
3 . The non-transitory computer readable medium of claim 1 , wherein the historical data is previously stored in the repository based on a previous request by the user.
4 . The non-transitory computer readable medium of claim 1 , wherein the relevant portion of the historical data includes a portion of the historical data that is labeled as originating from raw data received from the data source.
5 . The non-transitory computer readable medium of claim 1 , wherein the at least one data source is associated with an Application Programming Interface (API) and wherein the at least one action is interpreted as one or more API calls.
6 . The non-transitory computer readable medium of claim 1 , wherein the operations further comprise:
based on the determination that the supplemented data does not include the instruction, obtaining the formatted data from a second trained model.
7 . The non-transitory computer readable medium of claim 6 , wherein the first trained model and the second trained model are the same model.
8 . The non-transitory computer readable medium of claim 6 , wherein at least one of the at least one action identifier and the formatted data is based on a Retrieval-Augmented Generation (RAG) process associated with at least one data source.
9 . The non-transitory computer readable medium of claim 1 , wherein supplementing the raw data with the relevant portion of the historical data includes concatenating the raw data and the relevant portion of the historical data.
10 . The non-transitory computer readable medium of claim 1 , wherein the operations further comprise:
based on the determination that the supplemented data does not include the instruction, providing, to the user, a response to the original request.
11 . The non-transitory computer readable medium of claim 10 , wherein the response is based on an output from a second trained model.
12 . The non-transitory computer readable medium of claim 1 , wherein the first trained model is a large language model.
13 . The non-transitory computer readable medium of claim 1 , wherein receiving the raw data from the data source includes fetching the raw data by performing the at least one action.
14 . A computer-implemented method for detecting and mitigating indirect instructions provided to a trained model, the method comprising:
receiving, at an orchestrator application associated with a first trained model, an original request from a user;
identifying historical data associated with one or more previous interactions with the user;
receiving, from the first trained model, at least one action identifier associated with raw data stored in at least one data source, the action identifier specifying at least one action to be performed using the at least one data source;
receiving the raw data from the at least one data source as a result of performing the at least one action based on the at least one action identifier;
supplementing the raw data with a relevant portion of the historical data to generate supplemented data;
determining, based on the supplemented data, whether the supplemented data includes an instruction;
based on a determination that the supplemented data does not include the instruction, storing formatted data, the original request, and the raw data in a repository, the formatted data being based on the raw data, the original request, and the historical data; and
based on a determination that the supplemented data includes the instruction, performing a control action.
15 . The method of claim 14 , wherein determining whether the supplemented data includes the instruction includes making the supplemented data available to an indirect instructions detection tool configured to analyze the supplemented data.
16 . The method of claim 15 , wherein the indirect instructions detection tool includes an additional trained model.
17 . The method of claim 14 , wherein the control action includes deleting at least a portion of the historical data previously stored in the repository.
18 . The method of claim 14 , wherein the control action includes preventing the instruction from being provided to a second trained model.
19 . The method of claim 14 , wherein the control action includes logging an indication of the instruction.
20 . The method of claim 14 , wherein the control action includes terminating a session of the user.
21 . The method of claim 14 , wherein the control action includes returning an error indication to the user.
22 . The method of claim 14 , wherein the at least one data source is associated with a history manager.
23 . The method of claim 14 , further comprising:
identifying a functionality provider associated with the trained model;
requesting, from the functionality provider, functionality descriptor information indicating at least one function provided by the functionality provider; and
determining whether the functionality descriptor information includes a manipulative instruction, the determination whether the functionality descriptor information includes the manipulative instruction being based on analysis of the functionality descriptor information.
24 . The method of claim 14 , wherein the at least one action identifier was identified by the first trained model based on the request and the historical data.
25 . The method of claim 14 , wherein determining whether the supplemented data includes an instruction includes analyzing the supplemented data.
26 . A non-transitory computer readable medium including instructions that, when executed by at least one processor, cause the at least one processor to perform operations for detecting and mitigating indirect instructions provided to a trained model, the operations comprising:
identifying, at an indirect instructions detector, an input including raw data, the raw data being retrieved from at least one data source based on an original request from a user;
identifying a relevant portion of historical data, the historical data being associated with one or more previous interactions with the user;
obtaining supplemented data based on the raw data and the relevant portion of the historical data;
analyzing the supplemented data to determine whether the supplemented data includes an instruction;
based on a determination that the supplemented data does not include the instruction, storing formatted data, the original request, and the raw data in a repository, the formatted data being based on the raw data, the original request, and the historical data; and
based on a determination that the supplemented data includes the instruction, causing a control action to be performed.