Systems and methods for neural network safety moderators
In view of the potential misuse of AI to disseminate dangerous knowledge, embodiments described herein provide an AI safety training framework with a data generation pipeline for generating datasets indicative of multiple different severity levels of potentially harmful LLM-generated text to identify and respond to a potentially harmful query, and/or moderate a potentially harmful LLM-generated text. In this way, an LLM moderator may be integrated as a safety layer within an AI system's architecture, acting as a “firewall” to filter responses generated by another AI chat engine.
1 . A method of training a neural network based language model for moderating safety content, the method comprising:
receiving, via a data interface, a user query and a corresponding neural network generated response;
generating, by one or more neural network based language models, a plurality of revised responses at multiple safety severity levels based on the corresponding neural network generated response and an instruction indicative of the multiple safety severity levels;
training, the one or more neural network based language models, using a first dataset comprising the user query and the plurality of revised responses at multiple safety severity levels to generate a plurality of updated responses at a requested safety severity level,
wherein the one or more neural network based language models include at least one generator model, and training the at least one generator model includes:
generating, in response to the user query and the instruction indicative of the multiple safety severity levels, one or more predicted responses at the multiple safety severity levels, respectively; and
updating weights and parameters of the generator model based on a loss comparing the one or more predicted responses and the plurality of revised responses at the multiple safety severity levels, respectively;
training, the one or more neural network based language models, using a second dataset comprising the user query, the plurality of updated responses at the requested safety severity level, to predict a severity level associated with a model-generated text; and
building an artificial intelligence (AI) safety moderator agent by deploying the trained one or more neural network based language models on a hardware platform to moderate the model-generated text that is determined to be harmful.
2 . The method of claim 1 , further comprising:
forming the second dataset by augmented responses at the multiple severity levels that are generated by the trained generator model.
3 . The method of claim 1 , wherein the training of the at least one moderator model comprises:
generating, in response to the user query, at least one response from the second dataset and an instruction describing the multiple safety severity levels, at least one of:
a first predicted binary classification indicating whether the at least response is safe;
a second predicted binary classification indicating whether the user query is safe; or
a predicted severity level associated with the at least one response; and
updating weights and parameters of the moderator model based on a first loss comparing the first predicted binary classification, the second predicted binary classification, the predicted severity level and ground-truth labels from the second dataset, respectively.
4 . The method of claim 3 , wherein the training the at least one moderator model further comprises:
generating, in response to the user query, at least one response of a first severity level from the second dataset and an instruction describing the multiple safety severity levels, a moderated version of the at least one response at a second severity level; and
further updating the weights and the parameters of the moderator model based on a second loss comparing the moderated version of the at least one response and a ground-truth response at the second severity level from the second dataset.
5 . The method of claim 3 , further comprising:
determining the first predicted binary classification is different from a corresponding ground-truth label in the second dataset; and
iteratively re-generating the plurality of revised responses at the multiple safety severity levels from the at least response to form the first dataset in a next training iteration.
6 . The method of claim 1 , further comprising:
in response to determining, by the AI safety moderator agent, that the model-generated text from another AI agent is harmful:
generating, by the AI safety moderator agent, an updated text at a reduced severity level based on an input of an original query, the model-generated text, and a description of the multiple safety severity levels; and
sending the updated text to the other AI agent in place of the model-generated text.
7 . The method of claim 6 , further comprising:
in response to determining that an amount of harmful model-generated texts from the other AI agent is greater than a threshold, sending an alert to the other AI agent thereby causing a disablement of the other AI agent through an application programming interface (API).
8 . A system for training a neural network based language model for moderating safety content, the method comprising, the system comprising:
a memory that stores a plurality of processor executable instructions;
a communication interface that receives a user query and a corresponding neural network generated response; and
one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising:
generating, by one or more neural network based language models, a plurality of revised responses at multiple safety severity levels based on the corresponding neural network generated response and an instruction indicative of the multiple safety severity levels;
training, the one or more neural network based language models, using a first dataset comprising the user query and the plurality of revised responses at multiple safety severity levels to generate a plurality of updated responses at a requested safety severity level,
wherein the one or more neural network based language models include at least one generator model, and training the at least one generator model includes:
generating, in response to the user query and the instruction indicative of the multiple safety severity levels, one or more predicted responses at the multiple safety severity levels, respectively; and
updating weights and parameters of the generator model based on a loss comparing the one or more predicted responses and the plurality of revised responses at the multiple safety severity levels, respectively;
training, the one or more neural network based language models, using a second dataset comprising the user query, the plurality of updated responses at the requested safety severity level, to predict a severity level associated with a model-generated text; and
building an artificial intelligence (AI) safety moderator agent by deploying the trained one or more neural network based language models on a hardware platform to moderate the model-generated text that is determined to be harmful.
9 . The system of claim 8 , wherein the operations further comprise:
forming the second dataset by augmented responses at the multiple severity levels that are generated by the trained generator model.
10 . The system of claim 8 , wherein the training of the at least one moderator model comprises:
generating, in response to the user query, at least one response from the second dataset and an instruction describing the multiple safety severity levels, at least one of:
a first predicted binary classification indicating whether the at least response is safe;
a second predicted binary classification indicating whether the user query is safe; or
a predicted severity level associated with the at least one response; and
updating weights and parameters of the moderator model based on a first loss comparing the first predicted binary classification, the second predicted binary classification, the predicted severity level and ground-truth labels from the second dataset, respectively.
11 . The system of claim 10 , wherein the operation of training the at least one moderator model further comprises:
generating, in response to the user query, at least one response of a first severity level from the second dataset and an instruction describing the multiple safety severity levels, a moderated version of the at least one response at a second severity level; and
further updating the weights and the parameters of the moderator model based on a second loss comparing the moderated version of the at least one response and a ground-truth response at the second severity level from the second dataset.
12 . The system of claim 11 , wherein the operations further comprise:
determining the first predicted binary classification is different from a corresponding ground-truth label in the second dataset; and
iteratively re-generating the plurality of revised responses at the multiple safety severity levels from the at least response to form the first dataset in a next training iteration.
13 . The system of claim 8 , wherein the operations further comprise:
in response to determining, by the AI safety moderator agent, that the model-generated text from another AI agent is harmful:
generating, by the AI safety moderator agent, an updated text at a reduced severity level based on an input of an original query, the model-generated text, and a description of the multiple safety severity levels; and
sending the updated text to the other AI agent in place of the model-generated text.
14 . The system of claim 13 , wherein the operations further comprise:
in response to determining that an amount of harmful model-generated texts from the other AI agent is greater than a threshold, sending an alert to the other AI agent thereby causing a disablement of the other AI agent through an application programing interface (API).
15 . A non-transitory machine-readable medium comprising a plurality of machine-executable instructions for training a neural network based language model for moderating safety content, which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising:
receiving, via a data interface, a user query and a corresponding neural network generated response;
generating, by one or more neural network based language models, a plurality of revised responses at multiple safety severity levels based on the corresponding neural network generated response and an instruction indicative of the multiple safety severity levels;
training, the one or more neural network based language models, using a first dataset comprising the user query and the plurality of revised responses at multiple safety severity levels to generate a plurality of updated responses at a requested safety severity level,
wherein the one or more neural network based language models include at least one generator model, and training the at least one generator model includes:
generating, in response to the user query and the instruction indicative of the multiple safety severity levels, one or more predicted responses at the multiple safety severity levels, respectively; and
updating weights and parameters of the generator model based on a loss comparing the one or more predicted responses and the plurality of revised responses at the multiple safety severity levels, respectively;
training, the one or more neural network based language models, using a second dataset comprising the user query, the plurality of updated responses at the requested safety severity level, to predict a severity level associated with a model-generated text; and
building an artificial intelligence (AI) safety moderator agent by deploying the trained one or more neural network based language models on a hardware platform to moderate the model-generated text that is determined to be harmful.
16 . The non-transitory machine-readable medium of claim 15 , wherein the operations further comprising:
in response to determining, by the AI safety moderator agent, that the model-generated text from another AI agent is harmful:
generating, by the AI safety moderator agent, an updated text at a reduced severity level based on an input of an original query, the model-generated text, and a description of the multiple safety severity levels; and
sending the updated text to the other AI agent in place of the model-generated text.
17 . The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise:
in response to determining that an amount of harmful model-generated texts from the other AI agent is greater than a threshold, sending an alert to the other AI agent thereby causing a disablement of the other AI agent through an application programming interface (API).
18 . The non-transitory machine-readable medium of claim 15 , wherein the training of the at least one moderator model comprises:
generating, in response to the user query, at least one response from the second dataset and an instruction describing the multiple safety severity levels, at least one of:
a first predicted binary classification indicating whether the at least response is safe;
a second predicted binary classification indicating whether the user query is safe; or
a predicted severity level associated with the at least one response; and
updating weights and parameters of the moderator model based on a first loss comparing the first predicted binary classification, the second predicted binary classification, the predicted severity level and ground-truth labels from the second dataset, respectively;
generating, in response to the user query, at least one response of a first severity level from the second dataset and an instruction describing the multiple safety severity levels, a moderated version of the at least one response at a second severity level; and
further updating the weights and the parameters of the moderator model based on a second loss comparing the moderated version of the at least one response and a ground-truth response at the second severity level from the second dataset;
determining the first predicted binary classification is different from a corresponding ground-truth label in the second dataset; and
iteratively re-generating the plurality of revised responses at the multiple safety severity levels from the at least response to form the first dataset in a next training iteration.
19 . The non-transitory machine-readable medium of claim 15 , wherein the operations further include:
forming the second dataset by augmented responses at the multiple severity levels that are generated by the trained generator model.
20 . The non-transitory machine-readable medium of claim 18 , wherein the training the at least one moderator model further comprises:
generating, in response to the user query, at least one response of a first severity level from the second dataset and an instruction describing the multiple safety severity levels, a moderated version of the at least one response at a second severity level; and
further updating the weights and the parameters of the moderator model based on a second loss comparing the moderated version of the at least one response and a ground-truth response at the second severity level from the second dataset.