Utilizing a multi-layer security architecture for agent reinforcement fine-tuning of a machine learning model
The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing a reinforcement fine-tuning pipeline to fine tune an agent machine learning (ML) model utilizing user-defined grader endpoints and/or external tools with a multi-layer security architecture. For example, the disclosed systems can utilize a multi-layer security architecture to filter incoming training datasets, perform content refusal checks, chain-of-thought leak detections, and/or governance oversights prior to training the agent ML model, during active training of the agent ML model using external tools and/or grader models, and/or during post-training of the agent ML model (prior to releasing a fine-tuned snapshot of the model). In addition, the disclosed systems can generate stateful trajectory rollouts to associate training trajectories of the agent machine learning model to unique identifiers to facilitate multiple environments and/or tasks to run concurrently while maintaining consistency across external tool calls and grading interactions during training.
1 . A computer-implemented method comprising:
generating, for a prompt received at a generative artificial intelligence (AI) system, a stateful training trajectory for an agent model of the generative AI system, wherein the stateful training trajectory comprises a trajectory identifier and a trajectory metadata constrained within a trajectory environment;
generating, utilizing the agent model to process the prompt as part of the stateful training trajectory within the trajectory environment, an agent output by calling a digital tool through a secure proxy architecture supporting encrypted communication between the agent model and the digital tool, the secure proxy architecture comprising:
one or more training clusters on which the agent model executes; and
a unified cluster boundary, separate from the one or more training clusters, configured to receive tool-call requests from the one or more training clusters and to communicate with an external endpoint of the digital tool based on the tool-call requests;
determining, from a grader model, a generative reward corresponding to the agent output; and
updating parameters of the agent model based on the generative reward.
2 . The computer-implemented method of claim 1 , further comprising:
calling the digital tool through the secure proxy architecture to utilize an external digital tool from a third-party system to receive a third-party tool output; and
generating the agent output utilizing the agent model with the third-party tool output.
3 . The computer-implemented method of claim 2 , further comprising utilizing one or more moderation models of the generative AI system to analyze the third-party tool output of the external digital tool for a security vulnerability of the agent model.
4 . The computer-implemented method of claim 3 , further comprising:
upon receiving the third-party tool output, calling, through the secure proxy architecture, an additional external digital tool from the third-party system to receive an additional third-party tool output;
utilizing the one or more moderation models of the generative AI system to analyze the additional third-party tool output of the additional external digital tool for an additional security vulnerability of the agent model; and
generating the agent output utilizing the agent model with the third-party tool output and the additional third-party tool output.
5 . The computer-implemented method of claim 2 , further comprising utilizing one or more moderation models of the generative AI system to analyze a call to the external digital tool for a security vulnerability of the agent model, wherein the security vulnerability comprises a chain-of-thought leak.
6 . The computer-implemented method of claim 1 , further comprising determining, from the grader model, the generative reward by utilizing an external grader model from a third-party system.
7 . The computer-implemented method of claim 6 , further comprising, at each external grader model configuration variation, validating the external grader model within the trajectory environment by utilizing a chain-of-thought (CoT) classifier model.
8 . The computer-implemented method of claim 1 , further comprising:
generating an updated agent model snapshot by updating the parameters of the agent model;
generating a final agent output utilizing the updated agent model snapshot with the prompt; and
utilizing one or more moderation models of the generative AI system to analyze the final agent output for a security vulnerability of the updated agent model snapshot, wherein the security vulnerability comprises a chain-of-thought leak or a prohibited content item.
9 . The computer-implemented method of claim 8 , further comprising providing access to the updated agent model snapshot to a third-party system corresponding to the prompt based the agent output lacking the security vulnerability.
10 . The computer-implemented method of claim 1 , further comprising utilizing the stateful training trajectory to fine tune the agent model over multiple training trajectories comprising calls to one or more external digital tools.
11 . A system comprising:
at least one processor; and
a non-transitory computer readable medium comprising instructions that, when executed by the at least one processor, cause the system to:
receive a training prompt for an agent model of a generative artificial intelligence (AI) system from a user of a third-party system;
receive a third-party tool output by utilizing the training prompt with the agent model to call an external digital tool through a secure proxy architecture comprising:
one or more training clusters on which the agent model executes; and
a unified cluster boundary, separate from the one or more training clusters, configured to receive tool-call requests from the one or more training clusters and to communicate with an external endpoint of the external digital tool based on the tool-call requests;
determine a security vulnerability check for the third-party tool output utilizing one or more moderation models of the generative AI system; and
based on the security vulnerability check indicating a passed third-party tool output:
generate, utilizing the agent model, an agent output based on the third-party tool output;
update parameters of the agent model based on a generative reward corresponding to the agent output to generate an updated agent model snapshot; and
provide, to the user, access to the updated agent model snapshot.
12 . The system of claim of 11 , further comprising instructions that, when executed by the at least one processor, cause the system to determine the generative reward from an external grader model from the third-party system.
13 . The system of claim 11 , further comprising instructions that, when executed by the at least one processor, cause the system to provide, for display within a graphical user interface of a client device, aggregated training information, runtime metrics, or security validation results for the agent model.
14 . The system of claim of 11 , further comprising instructions that, when executed by the at least one processor, cause the system to utilize the one or more moderation models of the generative AI system to analyze the call to the external digital tool for a security vulnerability of the agent model, wherein the security vulnerability comprises a chain-of-thought leak.
15 . The system of claim of 11 , further comprising instructions that, when executed by the at least one processor, cause the system to:
generating a final agent output utilizing the updated agent model snapshot with the training prompt;
utilizing one or more moderation models of the generative AI system to analyze the final agent output for a security vulnerability of the updated agent model snapshot, wherein the security vulnerability comprises a chain-of-thought leak or a prohibited content item; and
provide access to the updated agent model snapshot based on the agent output lacking the security vulnerability.
16 . A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to:
receive a training prompt for an agent model of a generative artificial intelligence (AI) system from a user from a third-party system;
receive a third-party tool output by utilizing the training prompt with the agent model to call an external digital tool through a secure proxy architecture comprising:
one or more training clusters on which the agent model executes; and
a unified cluster boundary, separate from the one or more training clusters, configured to receive tool-call requests from the one or more training clusters and to communicate with an external endpoint of the external digital tool based on the tool-call requests;
determine a security vulnerability check for the third-party tool output utilizing one or more moderation models of the generative AI system; and
selectively fine tune parameters of the agent model utilizing the third-party tool output based on the security vulnerability check of the third-party tool output.
17 . The non-transitory computer-readable medium of claim 16 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to:
determine a passed third-party tool output for the security vulnerability check by utilizing the one or more moderation models of the generative AI system to analyze the third-party tool output of the external digital tool to identify a lack of a security vulnerability for the agent model; and
in response to the passed third-party tool output, selectively fine tune parameters of the agent model by modifying the parameters of the agent model based on the third-party tool output.
18 . The non-transitory computer-readable medium of claim 17 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to:
generate, utilizing the agent model, an agent output based on the third-party tool output;
update parameters of the agent model based on a generative reward corresponding to the agent output to generate an updated agent model snapshot; and
provide, to the user, access to the updated agent model snapshot.
19 . The non-transitory computer-readable medium of claim 16 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to:
determine a failed third-party tool output for the security vulnerability check by utilizing the one or more moderation models of the generative AI system to analyze the third-party tool output of the external digital tool to identify a security vulnerability for the agent model; and
in response to the passed third-party tool output, selectively fine tune parameters of the agent model by terminating training of the agent model.
20 . The non-transitory computer-readable medium of claim 16 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to utilize the one or more moderation models of the generative AI system to analyze the call to the external digital tool for a security vulnerability of the agent model.