Secure systems of guardrails for securing the use of large language models (LLMs)
The present disclosure includes computer-implemented methods of guardrails for securely using large language models (LLMs). The method comprises monitoring user data flow using an application programming interface (API) and receiving an administrative policy from an administration communication interface. The method involves dynamically applying a plurality of LLM input inspectors to LLM input data. The application of the plurality of LLM input inspectors is based on the administration policy. The dynamic application of the plurality of LLM input inspectors is in sequence for latency optimization. The plurality of LLM input inspectors serve as LLM input guardrails for a plurality of secure deployed large language models (LLMs). The plurality of LLM input inspectors are configured by the administrative policy and validate the LLM input data to validated LLM input data based on the administration policy. Additionally, the method comprises dynamically applying a plurality of LLM output inspectors to LLM output data.
1 . A computer-implemented method of guardrails for securely using large language models (LLMs), the computer-implemented method comprising:
monitoring user data flow using an application programming interface (API) that intercepts network communications before reaching external LLM services, the application programming interface (API) receiving data from one or more of: a proxy, a client, a connector, an agent, a third-party application, a browser plug-in, or a device, the user data flow comprising LLM input data and LLM output data;
receiving an administrative policy from an administration communication interface;
executing a plurality of LLM input inspectors to the LLM input data based on the administrative policy, the plurality of LLM input inspectors comprising binary input guardrails and modifying input guardrails, the modifying input guardrails changing the LLM input data, the executing the plurality of LLM input inspectors being performed in a linear order comprising first applying the binary input guardrails in parallel and second applying the modifying input guardrails for latency optimization, the plurality of LLM input inspectors being LLM input guardrails for a plurality of secure deployed large language models (LLMs) comprising LLMs deployed with access controls and data encryption, the plurality of LLM input inspectors being configured by the administrative policy and validating the LLM input data based on the administrative policy, the validating comprising determining the LLM input data complies with the administrative policy;
analyzing the LLM input data using the plurality of LLM input inspectors;
determining the LLM input data violates an LLM input setting of the administrative policy based on the analyzing the LLM input data using the plurality of LLM input inspectors;
blocking the LLM input data from processing by the plurality of secure deployed large language models (LLMs) based on the determining the LLM input data violates the LLM input setting of the administrative policy;
executing a plurality of LLM output inspectors to the LLM output data based on the administrative policy, the plurality of LLM output inspectors comprising binary output guardrails and modifying output guardrails, the modifying output guardrails changing the LLM output data, the executing the plurality of LLM output inspectors being performed in a linear order comprising first applying the binary output guardrails in parallel and second applying the modifying output guardrails for latency optimization, the plurality of LLM output inspectors being LLM output guardrails for the plurality of secure deployed large language models (LLMs), the plurality of LLM output inspectors being configured by the administrative policy and validating the LLM output data based on the administrative policy, the validating comprising determining the LLM output data complies with the administrative policy;
analyzing the LLM output data using the plurality of LLM output inspectors;
determining the LLM output data violates an LLM output setting of the administrative policy based on the analyzing the LLM output data using the plurality of LLM output inspectors; and
blocking the LLM output data based on the determining the LLM output data violates the LLM output setting of the administrative policy.
2 . The computer-implemented method of claim 1 , wherein the plurality of LLM input inspectors comprise one or more of: a prompt injection filter, a model drift filter, a Personally Identifiable Information (PII) redacting filter, a banned topics filter, a banned string filter, a bias filter, a source code filter, a toxicity filter, a source tagging filter, a use cases filter, a token count filter, an encryption filter, a security filter, an API filter, or a third-party developed filter.
3 . The computer-implemented method of claim 1 , wherein the plurality of LLM output inspectors comprise one or more of: a hallucination filter, a prompt injection filter, a malicious URLs filter, a source code filter, a model drift filter, a refutation filter, a Personally Identifiable Information (PII) un-redacting filter, a banned topics filter, an on topic filter, an allowed topics filter, a banned string filter, a bias filter, a toxicity filter, a source tagging filter, a use cases filter, a token count filter, an encryption filter, a relevance filter, an anonymization filter, a sentiment filter, an access control filter, a regex filter, a decryption filter, or a third-party developed filter.
4 . The computer-implemented method of claim 1 ,
wherein the LLM input data comprises Personally Identifiable Information (PII);
wherein the plurality of LLM input inspectors comprise a PII redacting filter configured to redact the PII by replacing it with a tag before processing by the plurality of secure deployed LLMs; and
wherein the plurality of LLM output inspectors comprise a Personally Identifiable Information (PII) un-redacting filter, the Personally Identifiable Information (PII) un-redacting filter un-redacting the Personally Identifiable Information (PII) after the processing of the redacted Personally Identifiable Information (PII) by the plurality of secure deployed large language models (LLMs), the un-redacting replacing the tag with the Personally Identifiable Information (PII).
5 . The computer-implemented method of claim 1 ,
wherein the LLM input data comprises a prompt; and
wherein the plurality of LLM input inspectors comprise a classifier input inspector, the classifier input inspector determining an intention of a prompt.
6 . The computer-implemented method of claim 5 , further comprising:
routing the LLM input data to an approved secure deployed LLM of the plurality of secure deployed LLMs using a routing engine, the routing engine receiving the intention of the prompt and routing the LLM input data to the approved secure deployed LLM based on the intention of the prompt.
7 . The computer-implemented method of claim 5 , further comprising:
determining a security risk score for the prompt;
comparing the security risk score for the prompt to a security threshold; and
performing an action on the LLM input data based on the comparing the security risk score for the prompt to the security threshold.
8 . The computer-implemented method of claim 7 ,
wherein the performing the action on the LLM input data comprises one or more of:
blocking the LLM input data from processing by the plurality of secure deployed large language models (LLMs);
allowing the LLM input data for processing by the plurality of secure deployed large language models (LLMs); and
generating a warning, logging, or
calling a third-party application programming interface (API).
9 . The computer-implemented method of claim 1 ,
wherein the LLM input data comprises one of confidential information or non-confidential information;
wherein the plurality of LLM input inspectors comprise a classifier input inspector, the classifier input inspector determining whether the LLM input data comprises the confidential information or the non-confidential information; and
routing the LLM input data using a routing engine to:
(i) a private secure deployed large language model (LLM) of the plurality of secure deployed large language models (LLMs) when the LLM input data comprises confidential information, or
(ii) a public deployed large language model (LLM) of the plurality of secure deployed large language models (LLMs) when the LLM input data comprises non-confidential information,
wherein the routing engine receives the determination from the classifier input inspector.
10 . The computer-implemented method of claim 1 ,
wherein the plurality of LLM input inspectors are applied to the LLM input data to provide metadata regarding the LLM input data, and the metadata comprising one or more of: a request type, a topic, a subtopic, an intention, a risk metric, a risk type, or a concern analysis.
11 . The computer-implemented method of claim 1 ,
wherein the administration communication interface comprises a dashboard interface and one or more of: a policy engine, an observability engine, a compliance engine, an audit engine, a best practices engine, a logging engine, a risk scoring engine, a third-party application tracker, a recommendation engine, a reporting engine, or an alert engine.
12 . The computer-implemented method of claim 1 ,
wherein the LLM input data comprises a prompt, further comprising receiving one or more of: an intention of the prompt, a request type of the prompt, a topic of the prompt, a subtopic of the prompt, a risk metric of the prompt, a risk type of the prompt, or a concern analysis of the prompt from a classifier input inspector;
wherein the executing the plurality of LLM input inspectors to the LLM input data is further based on the one or more of: the intention of the prompt, the request type of the prompt, the topic of the prompt, the subtopic of the prompt, the risk metric of the prompt, the risk type of the prompt, or the concern analysis of the prompt.
13 . A computer-implemented method of guardrails for securely using large language models (LLMs), the computer-implemented method comprising:
monitoring user data flow using an application programming interface (API) that intercepts network communications before reaching external LLM services, the application programming interface (API) receiving data from one or more of: a proxy, a client, a connector, an agent, a third-party application, a browser plug-in, or a device, the user data flow comprising LLM input data and LLM output data, the LLM input data comprising a prompt;
receiving an administrative policy from an administration communication interface;
receiving from a classifier input inspector one or more of: an intention of the prompt, a request type of the prompt, a topic of the prompt, a subtopic of the prompt, a risk metric of the prompt, a risk type of the prompt, or a concern analysis of the prompt, the intention comprising one or more of: information retrieval, code generation, content creation, data analysis, question answering, or document summarization, and the risk metric being a numerical value calculated based on factors comprising presence of injection patterns, toxicity level, banned topic detection, and personally identifiable information content;
executing a plurality of LLM input inspectors to the LLM input data based on the administrative policy and the one or more of: the intention of the prompt, the request type of the prompt, the topic of the prompt, the subtopic of the prompt, the risk metric of the prompt, the risk type of the prompt, or the concern analysis of the prompt, the plurality of LLM input inspectors comprising binary input guardrails and modifying input guardrails, the modifying input guardrails changing the LLM input data, the executing the plurality of LLM input inspectors being performed in a linear order comprising first applying the binary input guardrails in parallel and second applying the modifying input guardrails for latency optimization, the plurality of LLM input inspectors being LLM input guardrails for a plurality of secure deployed large language models (LLMs) comprising LLMs deployed with access controls and data encryption, the plurality of LLM input inspectors being configured by the administrative policy and validating the LLM input data based on the administrative policy and the intention of the prompt, the validating comprising determining the LLM input data complies with the administrative policy;
analyzing the LLM input data using the plurality of LLM input inspectors;
determining the LLM input data violates an LLM input setting of the administrative policy based on the analyzing the LLM input data using the plurality of LLM input inspectors;
blocking the LLM input data from processing by the plurality of secure deployed large language models (LLMs) based on the determining the LLM input data violates the LLM input setting of the administrative policy;
executing a plurality of LLM output inspectors to the LLM output data based on the administrative policy and the one or more of: the intention of the prompt, the request type of the prompt, the topic of the prompt, the subtopic of the prompt, the risk metric of the prompt, the risk type of the prompt, or the concern analysis of the prompt, the plurality of LLM output inspectors comprising binary output guardrails and modifying output guardrails, the modifying output guardrails changing the LLM output data, the executing the plurality of LLM output inspectors being performed in a linear order comprising first applying the binary output guardrails in parallel and second applying the modifying output guardrails for latency optimization, the plurality of LLM output inspectors being LLM output guardrails for the plurality of secure deployed large language models (LLMs), the plurality of LLM output inspectors being configured by the administrative policy and validating the LLM output data based on the administrative policy and the intention of the prompt, the validating comprising determining the LLM output data complies with the administrative policy;
analyzing the LLM output data using the plurality of LLM output inspectors;
determining the LLM output data violates an LLM output setting of the administrative policy based on the analyzing the LLM output data using the plurality of LLM output inspectors; and
blocking the LLM output data based on the determining the LLM output data violates the LLM output setting of the administrative policy.
14 . A computer-implemented method of guardrails for securely using large language models (LLMs), the computer-implemented method comprising:
monitoring user data flow using an application programming interface (API), the application programming interface (API) receiving data from one or more of: a proxy, a client, a connector, an agent, a third-party application, a browser plug-in, or a device, the user data flow comprising LLM input data and LLM output data, the LLM input data comprising a prompt;
receiving an administrative policy from an administration communication interface;
receiving from a classifier input inspector one or more of: an intention of the prompt, a request type of the prompt, a topic of the prompt, a subtopic of the prompt, a risk metric of the prompt, a risk type of the prompt, or a concern analysis of the prompt;
executing a plurality of LLM input inspectors to the LLM input data based on the administrative policy and the one or more of: the intention of the prompt, the request type of the prompt, the topic of the prompt, the subtopic of the prompt, the risk metric of the prompt, the risk type of the prompt, or the concern analysis of the prompt, the plurality of LLM input inspectors comprising binary input guardrails and modifying input guardrails, the modifying input guardrails changing the LLM input data, the executing the plurality of LLM input inspectors being performed in a linear order comprising first applying the binary input guardrails in parallel and second applying the modifying input guardrails for latency optimization, the plurality of LLM input inspectors being LLM input guardrails for a plurality of secure deployed large language models (LLMs) comprising LLMs deployed with access controls and data encryption, the plurality of LLM input inspectors being configured by the administrative policy and validating the LLM input data based on the administrative policy and the intention of the prompt, the validating comprising determining the LLM input data complies with the administrative policy;
analyzing the LLM input data using the plurality of LLM input inspectors;
determining the LLM input data violates an LLM input setting of the administrative policy based on the analyzing the LLM input data using the plurality of LLM input inspectors;
blocking the LLM input data from processing by the plurality of secure deployed large language models (LLMs) based on the determining the LLM input data violates the LLM input setting of the administrative policy;
executing a plurality of LLM output inspectors to the LLM output data based on the administrative policy and the one or more of: the intention of the prompt, the request type of the prompt, the topic of the prompt, the subtopic of the prompt, the risk metric of the prompt, the risk type of the prompt, or the concern analysis of the prompt, the plurality of LLM output inspectors comprising binary output guardrails and modifying output guardrails, the modifying output guardrails changing the LLM output data, the executing the plurality of LLM output inspectors being performed in a linear order comprising first applying the binary output guardrails in parallel and second applying the modifying output guardrails for latency optimization, the plurality of LLM output inspectors being LLM output guardrails for the plurality of secure deployed large language models (LLMs), the plurality of LLM output inspectors being configured by the administrative policy and validating the LLM output data based on the administrative policy and the intention of the prompt, the validating comprising determining the LLM output data complies with the administrative policy;
analyzing the LLM output data using the plurality of LLM output inspectors;
determining the LLM output data violates an LLM output setting of the administrative policy based on the analyzing the LLM output data using the plurality of LLM output inspectors; and
blocking the LLM output data based on the determining the LLM output data violates the LLM output setting of the administrative policy.
15 . The computer-implemented method of claim 1 , further comprising one or more of: generating a warning, logging, or calling a third-party application programming interface (API) based on the validating the LLM output data.