Retraining a large language model

1 . A system, comprising:

a processor configured to:

receive a plurality of indications of a plurality of different problems associated with a plurality of responses generated by a first artificial intelligence (AI) language model;

generate by the first AI language model a plurality of modified responses for the plurality of responses;

provide to a second AI language model the plurality of modified responses and some or all of a turn-based conversation that included a corresponding response of the plurality of responses, wherein the second AI language model ranks the plurality of modified responses based on how well each of the plurality of modified responses responds to a corresponding problem of the plurality of different problems that was indicated and provides the ranking to the first AI language model;

generate a synthetic dataset by combining the plurality of modified responses and the some or all of a turn-based conversation that included the response based on the ranking, wherein the synthetic dataset includes different proportions of data associated with different problems of the plurality of different problems; and

retrain the first AI language model using the synthetic dataset; and

a memory coupled to the processor and configured to provide the processor with instructions.

2 . The system of claim 1 , wherein the processor is further configured to perform the turn-based conversation with a client device.

3 . The system of claim 1 , wherein the turn-based conversation comprises a narrative.

4 . The system of claim 1 , wherein the processor is further configured to store the turn-based conversation and a received indication of the plurality of indications in a database.

5 . The system of claim 1 , wherein a problem of the plurality of different problems is one of the following: memory problems, repetition problems, conversation stagnation, or safety problems.

6 . The system of claim 1 , wherein the second AI language model is configured to provide feedback associated with the plurality of modified responses.

7 . The system of claim 1 , wherein the processor is further configured to re-train the first AI language model using one or more synthetic datasets, wherein the one or more synthetic datasets includes some or all of the turn-based conversation.

8 . The system of claim 1 , wherein the processor is further configured to re-train the first AI chatbot language model using one or more synthetic datasets, wherein each of the one or more synthetic datasets comprises proportions of training data associated with different problems.

9 . The system of claim 1 , wherein an indication of the plurality of indications includes a text description of the problem associated with a response of the plurality of responses.

10 . The system of claim 1 , wherein a problem of the plurality of different problems associated with a response of the plurality of responses is chosen by a user.

11 . The system of claim 1 , wherein the processor is further configured to determine that a time condition has been satisfied, wherein the first AI language model is retrained in response to a determination that the time condition has been satisfied.

12 . The system of claim 1 , wherein the synthetic dataset includes unproblematic data, wherein the unproblematic data includes a plurality of unproblematic responses.

13 . A method, comprising:

receiving a plurality of indications of a plurality of different problems associated with a plurality of responses generated by a first artificial intelligence (AI) language model;

generating by the first AI language model a plurality of modified responses for the plurality of responses;

providing to a second AI language model the plurality of modified responses and some or all of a turn-based conversation that included a corresponding response of the plurality of responses, wherein the second AI language model ranks the plurality of modified responses based on how well each of the plurality of modified responses responds to a corresponding problem of the plurality of different problems that was indicated and provides the ranking to the first AI language model;

generating a synthetic dataset by combining the plurality of modified responses and the some or all of a turn-based conversation that included the response based on the ranking, wherein the synthetic dataset includes different proportions of data associated with different problems of the plurality of different problems; and

retraining the first AI language model using the synthetic dataset.

14 . The method of claim 13 , further comprising performing the turn-based conversation with a client device.

15 . The method of claim 14 , further comprising storing the turn-based conversation and a received indication of the plurality of indications in a database.

16 . The method of claim 14 , wherein a problem of the plurality of different problems is one of the following: memory problems, repetition problems, conversation stagnation, or safety problems.

17 . The method of claim 14 , wherein the second AI language model is configured to provide feedback associated with the plurality of modified responses.

18 . A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:

receiving a plurality of indications of a plurality of different problems associated with a plurality of responses generated by a first artificial intelligence (AI) language model;

generating by the first AI language model a plurality of modified responses for the plurality of responses;

providing to a second AI language model the plurality of modified responses and some or all of a turn-based conversation that included a corresponding response of the plurality of responses, wherein the second AI language model ranks the plurality of modified responses based on how well each of the plurality of modified responses responds to a corresponding problem of the plurality of different problems that was indicated and provides the ranking to the first AI language model;

generating a synthetic dataset by combining the plurality of modified responses and the some or all of a turn-based conversation that included the response based on the ranking, wherein the synthetic dataset includes different proportions of data associated with different problems of the plurality of different problems; and

retraining the first AI language model using the synthetic dataset.