Optimizing retrieval-augmented generation systems through enhanced document selection

1 . A method comprising:

applying a document ranking layer of a document selection large language model (LLM) to a first document list corresponding to a training dataset to obtain a ranked document list, wherein the first document list comprises a first plurality of reference documents;

selecting a subset of the first plurality of reference documents from the ranked document list to obtain a first document subset of the first plurality of reference documents;

processing a first user prompt and the first document subset with a field LLM to generate a first answer, wherein the first user prompt corresponds to the training dataset;

ranking, by a ranking LLM, the first answer with an answer score;

ranking, by the ranking LLM, the first document subset to obtain a ranked document subset;

calculating a loss function of a preference optimization layer of the document selection LLM based on the answer score;

updating at least one training parameter of a foundation model of the document selection LLM based on the loss function of the preference optimization layer;

retrieving, for a second user prompt, a second document list comprising a second plurality of reference documents;

processing, with the document selection LLM after updating the at least one training parameter, the second document list to obtain a second document subset of the second plurality of reference documents, wherein the second document subset is ordered in decreasing relevance; and

generating a second answer with the field LLM processing the second user prompt using the second document subset.

2 . The method of claim 1 , further comprising:

applying the document ranking layer of the document selection LLM to the first document list corresponding to the training dataset to obtain the ranked document list, wherein the document ranking layer ranks the first document list, the ranking comprising:

ranking a reference document of the first plurality of reference documents of the first document list based on an aggregate score of an aggregate score set of the training dataset, wherein the aggregate score corresponds to the reference document; and

reordering the ranked document list based on the rank of the reference document.

3 . The method of claim 1 , further comprising:

ranking the first answer, wherein ranking further comprises:

receiving, by the ranking LLM as answer parameters, (i) the first answer, and (ii) a preferred answer corresponding to the training dataset, along with an answer ranking instruction to rank the first answer based on a similarity of the first answer to the preferred answer,

processing, by the ranking LLM, the answer ranking instruction and the answer parameters to obtain the answer score, and

assigning, by the ranking LLM, the answer score to the first answer.

4 . The method of claim 1 , further comprising:

ranking the first document subset, wherein ranking further comprises:

receiving, by the ranking LLM as document parameters, (i) the first document subset and (ii) a preferred answer corresponding to the training dataset, along with a document ranking instruction to rank each reference document of the first document subset of the first plurality of reference documents based on a relevance of the reference document to the preferred answer,

processing, by the ranking LLM, the document ranking instruction and the document parameters to obtain a document ranking score,

assigning, by the ranking LLM, the document ranking score to each reference document of the first document subset of the first plurality of reference documents of the first document subset,

reordering the first document subset based on document ranking scores corresponding to each reference document of the first document subset of the first plurality of reference documents, to obtain the ranked document subset; and

outputting, by the ranking LLM, the ranked document subset.

5 . The method of claim 1 , further comprising:

selecting a plurality of subsets from the ranked document list to obtain a plurality of document subsets;

processing the first user prompt corresponding to the training dataset, and each document subset of the plurality of document subsets, by the field LLM to generate a plurality of answers corresponding to the plurality of document subsets;

ranking, by the ranking LLM, the plurality of answers to obtain a plurality of ranked answers, wherein each answer of the plurality of answers is assigned a corresponding answer score; and

ranking, by the ranking LLM, for each document subset of the plurality of document subsets, the first document subset of the first plurality of reference documents, wherein each reference document of the first document subset is assigned a document ranking score, to obtain a plurality of ranked document subsets.

6 . The method of claim 1 , further comprising:

retrieving, by a retrieval optimization engine, a past query structure from a data repository stored on a physical storage device, the past query structure comprising a past user prompt, and a past preferred answer;

retrieving, by a document retrieval application, from a document store in the data repository, a new document list comprising a plurality of new reference documents based on the past user prompt;

processing, by the field LLM, the past user prompt and the new document list to obtain a new answer based on the new document list;

ranking, by the ranking LLM, the new answer with a new answer ranking score; and

ranking, by the ranking LLM, the plurality of new reference documents of the new document list with a corresponding plurality of new document ranking scores.

7 . The method of claim 6 , further comprising:

generating a new aggregate score set corresponding to the new document list, comprising:

generating a new aggregate score corresponding to a new reference document of the plurality of new reference documents of the new document list, wherein the new aggregate score is a new document ranking score of the corresponding plurality of new document ranking scores corresponding to the new reference document of the plurality of new reference documents of the new document list, aggregated with the new answer ranking score, and

adding the new aggregate score to the new aggregate score set;

adding the past user prompt, the past preferred answer, the new document list, and the new aggregate score set to a new training dataset; and

adding the new training dataset to training data stored in the data repository.

8 . The method of claim 6 , wherein ranking the new answer further comprises:

receiving, by the ranking LLM, as new answer parameters, (i) the past user prompt, (ii) the past preferred answer, and (iii) the new answer, along with a new answer ranking instruction to rank the new answer based on a similarity of the new answer to the past preferred answer;

processing, by the ranking LLM, the new answer parameters and the new answer ranking instruction to generate a new answer ranking score; and

assigning, by the ranking LLM, the new answer ranking score to the new answer.

9 . The method of claim 6 , wherein ranking the plurality of new reference documents of the new document list further comprises:

receiving, by the ranking LLM as new document parameters, (i) the past preferred answer, and (ii) the new document list, along with a new document ranking instruction to rank each new reference document of the plurality of new reference documents of the new document list based on a relevance of the new reference document with respect to (i) the past user prompt and (ii) the past preferred answer;

processing, by the ranking LLM, the new document parameters and the new document ranking instruction to generate new document ranking scores corresponding respectively to each new reference document of the plurality of new reference documents of the new document list; and

assigning, by the ranking LLM, the new document ranking scores respectively to each new reference document of the plurality of new reference documents of the new document list.

10 . The method of claim 1 , further comprising:

processing, by the field LLM, the second user prompt and the second document subset to obtain a second answer corresponding to the second user prompt; and

transmitting the second answer as a response to a request comprising the second user prompt.

11 . A method comprising:

applying a document ranking layer of a document selection large language model (LLM) to a document list corresponding to a training dataset to obtain a ranked document list, wherein the document list comprises a plurality of reference documents;

selecting a subset of the plurality of reference documents from the ranked document list to obtain a document subset of the plurality of reference documents;

processing a user prompt and the document subset with a field LLM to generate an answer, wherein the user prompt corresponds to the training dataset;

ranking, by a ranking LLM, the answer with an answer score;

ranking, by the ranking LLM, the document subset to obtain a ranked document subset;

calculating a loss function of a preference optimization layer of the document selection LLM based on the answer score;

updating at least one training parameter of a foundation model of the document selection LLM based on the loss function of the preference optimization layer;

selecting a plurality of subsets from the ranked document list to obtain a plurality of document subsets;

processing the user prompt corresponding to the training dataset, and each document subset of the plurality of document subsets, by the field LLM to generate a plurality of answers corresponding to the plurality of document subsets;

ranking, by the ranking LLM, the plurality of answers to obtain a plurality of ranked answers, wherein each answer of the plurality of answers is assigned a corresponding answer score;

ranking, by the ranking LLM, for each document subset of the plurality of document subsets, the document subset of the plurality of reference documents, wherein each reference document of the first document subset is assigned a document ranking score, to obtain a plurality of ranked document subsets;

calculating the loss function of the preference optimization layer of the document selection LLM based on answer scores corresponding to each answer of the plurality of answers; and

updating the at least one training parameter of the foundation model of the document selection LLM, based on the loss function, wherein the at least one training parameter causes the foundation model to select the ranked document subset of the plurality of ranked document subsets corresponding to a ranked answer of the plurality of ranked answers with a highest answer score value as a preferred document subset.

12 . A system comprising:

at least one computer processor;

a data repository, in communication with the at least one computer processor, and stored on a physical storage device;

a retrieval optimization engine, executing on the at least one computer processor;

a document selection LLM, comprising a foundation model, a document ranking layer, and a preference optimization layer, and executing on the at least one computer processor; and

a ranking LLM, executing on the at least one compute processor,

wherein the retrieval optimization engine is configured to:

retrieve training data from the data repository, the training data comprising a training dataset,

cause the document selection LLM to apply the document ranking layer to a first document list corresponding to the training dataset to obtain a ranked document list, wherein the first document list comprises a first plurality of reference documents,

select a subset of the first plurality of reference documents from the ranked document list to obtain a first document subset of the first plurality of reference documents,

cause a field LLM to process a first user prompt and the first document subset to generate a first answer, wherein the first user prompt corresponds to the training dataset,

cause the ranking LLM to rank the first answer with an answer score,

cause the ranking LLM to rank the first document subset to obtain a ranked document subset,

cause the document selection LLM to calculate a loss function of the preference optimization layer, based on the answer score,

cause the document selection LLM to update at least one training parameter of the foundation model based on the loss function of the preference optimization layer,

retrieve, for a second user prompt, a second document list comprising a second plurality of reference documents,

cause the document selection LLM to process, after updating the at least one training parameter, the second document list to obtain a second document subset of the second plurality of reference documents, wherein the second document subset is ordered in decreasing relevance, and

generate a second answer with the field LLM processing the second user prompt using the second document subset.

13 . The system of claim 12 , wherein:

the document selection LLM is further configured to:

apply the document ranking layer to the first document list corresponding to the training dataset to rank the first document list, the ranking comprising:

ranking a reference document of the first plurality of reference documents of the first document list based on an aggregate score of an aggregate score set of the training dataset, wherein the aggregate score corresponds to the reference document, and

reordering the ranked document list based on the rank of the reference document.

14 . The system of claim 12 , wherein:

the ranking LLM is further configured to rank the first answer, the ranking further comprising:

receiving as answer parameters, (i) the first answer, and (ii) a preferred answer corresponding to the training dataset, along with an answer ranking instruction to rank the first answer based on a similarity of the first answer to the preferred answer,

processing the answer ranking instruction and the answer parameters to obtain the answer score, and

assigning the answer score to the first answer.

15 . The system of claim 12 , wherein:

the ranking LLM executing on the at least one computer processor is further configured to rank the first document subset, the ranking further comprising:

receiving as document parameters, (i) the first document subset and (ii) a preferred answer corresponding to the training dataset, along with a document ranking instruction to rank each reference document of the first document subset of the first plurality of reference documents based on a relevance of the reference document to the preferred answer,

processing the document ranking instruction and the document parameters to obtain a document ranking score,

assigning the document ranking score to each reference document of the first document subset of the first plurality of reference documents of the first document subset,

reordering the first document subset based on document ranking scores corresponding to each reference document of the first document subset of the first plurality of reference documents, to obtain the ranked document subset; and

outputting the ranked document subset.

16 . The system of claim 12 , wherein:

the document selection LLM is further configured to select a plurality of subsets from the ranked document list to obtain a plurality of document subsets; and

the retrieval optimization engine executing on the at least one computer processor is further configured to:

cause the field LLM executing on the at least one computer processor to process the first user prompt corresponding to the training dataset, and each document subset of the plurality of document subsets to generate a plurality of answers corresponding to the plurality of document subsets,

cause the ranking LLM executing on the at least one computer processor to rank the plurality of answers, to obtain a plurality of ranked answers, wherein each answer of the plurality of ranked answers is assigned a corresponding answer score, and

cause the ranking LLM executing on the at least one computer processor to rank, for each document subset of the plurality of document subsets, the first document subset of the first plurality of reference documents, wherein each reference document of the first document subset is assigned a document ranking score, to obtain a plurality of ranked document subsets.

17 . The system of claim 16 , wherein:

the document selection LLM is further configured to:

calculate the loss function of the preference optimization layer of the document selection LLM based on answer scores corresponding to each answer of the plurality of answers, and

update the at least one training parameter of the foundation model of the document selection LLM, based on the loss function, wherein the at least one training parameter causes the foundation model to select the ranked document subset of the plurality of ranked document subsets corresponding to a ranked answer of the plurality of ranked answers with a highest answer score value as a preferred document subset.

18 . The system of claim 12 , further comprising:

a document retrieval application, executing on the at least one computer processor;

and wherein:

the retrieval optimization engine executing on the at least one computer processor is further configured to retrieve a past query structure from the data repository, the past query structure comprising a past user prompt, and a past preferred answer,

the document retrieval application executing on the at least one computer processor is further configured to retrieve a new document list from a document store in the data repository, the new document list comprising a plurality of new reference documents based on the past user prompt,

the field LLM executing on the at least one computer processor is further configured to process the past user prompt and the new document list to obtain a new answer based on the new document list, and

the ranking LLM executing on the at least one computer processor is further configured to:

rank the new answer with a new answer ranking score, and

rank the plurality of new reference documents of the new document list with a corresponding plurality of new document ranking scores.

19 . The system of claim 18 , wherein:

the retrieval optimization engine is further configured to:

generate a new aggregate score set corresponding to the new document list, wherein generating comprises:

generating a new aggregate score corresponding to a new reference document of the plurality of new reference documents of the new document list, wherein the new aggregate score is a new document ranking score of the plurality of new document ranking scores corresponding to the new reference document of the plurality of new reference documents of the new document list, aggregated with the new answer ranking score, and

adding the new aggregate score to the new aggregate score set;

add the past user prompt, the past preferred answer, the new document list, and the new aggregate score set to a new training dataset; and

add the new training dataset to the training data.

20 . The system of claim 18 , wherein:

the ranking LLM executing on the at least one computer processor is further configured to:

receive, as new answer parameters, (i) the past user prompt, (ii) the past preferred answer, and (iii) the new answer, along with a new answer ranking instruction to rank the new answer based on a similarity of the new answer to the past preferred answer,

process the new answer parameters and the new answer ranking instruction to generate a new answer ranking score,

assign the new answer ranking score to the new answer,

receive, as new document parameters, (i) the past preferred answer, and (ii) the new document list, along with a new document ranking instruction to rank each new reference document of the plurality of new reference documents of the new document list based on a relevance score of the new reference document with respect to the past preferred answer,

process the new document parameters and the new document ranking instruction to generate new document ranking scores corresponding respectively to each new reference document of the plurality of new reference documents of the new document list; and

assign the new document ranking scores respectively to each new reference document of the plurality of new reference documents of the new document list.