Generative machine learning with retriever having reconfigurable sequence of rankers
A method includes obtaining an input query at a retriever model, where the retriever model includes a reconfigurable sequence of one or more rankers selected from among a plurality of rankers. Each ranker is configured to identify a specified number of information chunks relevant to the input query. The method also includes providing one or more of the information chunks from the retriever model to a generative model. The method further includes using the generative model to create a response to the input query, where the response is based on the one or more information chunks. The plurality of rankers includes a bi-encoder, a cross-encoder, and a large language model (LLM)-ranker.
1. A method comprising:
obtaining an input query at a retriever model, the retriever model comprising a reconfigurable sequence of one or more rankers selected from among a plurality of rankers, each ranker configured to identify a specified number of information chunks relevant to the input query;
providing one or more of the information chunks from the retriever model to a generative model;
using the generative model to create a response to the input query, the response based on the one or more information chunks; and
tuning the retriever model by determining the specified number of information chunks to be identified by each ranker in the reconfigurable sequence;
wherein the plurality of rankers comprises a bi-encoder, a cross-encoder, and a large language model (LLM)-ranker; and
wherein the specified number of information chunks to be identified by each ranker in the reconfigurable sequence is determined using a grid search.
2. The method of claim 1 , wherein the retriever model is configured to process different fields of information in the information chunks differently.
3. The method of claim 1 , wherein the retriever model is configured to dynamically select a size of the one or more information chunks provided to the generative model.
4. The method of claim 1 , further comprising:
providing the response to a user device associated with a user.
5. The method of claim 1 , wherein the generative model comprises a large language model.
6. The method of claim 1 , wherein the grid search used to determine the specified number of information chunks to be identified by each ranker in the reconfigurable sequence comprises:
selecting values for the specified number of information chunks to be identified by each ranker in the reconfigurable sequence using a pre-defined grid;
using the retriever model and the generative model to process a validation set that includes annotated document chunks based on the selected values;
evaluating an accuracy of a system that includes the retriever model and the generative model based on the processing of the validation set; and
simultaneously tuning the values for the specified number of information chunks to be identified by each ranker in the reconfigurable sequence based on results of the evaluating.
7. The method of claim 6 , wherein outputs generated by the retriever model and the generative model based on the validation set are compared to annotations of the annotated document chunks, the annotations treated as ground truths.
8. An apparatus comprising:
at least one processing device configured to:
provide an input query to a retriever model, the retriever model comprising a reconfigurable sequence of one or more rankers selected from among a plurality of rankers, each ranker configured to identify a specified number of information chunks relevant to the input query;
provide one or more of the information chunks from the retriever model to a generative model;
use the generative model to create a response to the input query, the response based on the one or more information chunks; and
determine the specified number of information chunks to be identified by each ranker in the reconfigurable sequence in order to tune the retriever model;
wherein the plurality of rankers comprises a bi-encoder, a cross-encoder, and a large language model (LLM)-ranker; and
wherein the at least one processing device is configured to determine the specified number of information chunks to be identified by each ranker in the reconfigurable sequence using a grid search.
9. The apparatus of claim 8 , wherein the retriever model is configured to process different fields of information in the information chunks differently.
10. The apparatus of claim 8 , wherein the retriever model is configured to dynamically select a size of the one or more information chunks provided to the generative model.
11. The apparatus of claim 8 , wherein the at least one processing device is further configured to provide the response to a user device associated with a user.
12. The apparatus of claim 8 , wherein the generative model comprises a large language model.
13. The apparatus of claim 8 , wherein, to determine the specified number of information chunks to be identified by each ranker in the reconfigurable sequence using the grid search, the at least one processing device configured to:
select values for the specified number of information chunks to be identified by each ranker in the reconfigurable sequence using a pre-defined grid;
use the retriever model and the generative model to process a validation set that includes annotated document chunks based on the selected values;
evaluate an accuracy of a system that includes the retriever model and the generative model based on the processing of the validation set; and
simultaneously tune the values for the specified number of information chunks to be identified by each ranker in the reconfigurable sequence based on results of the evaluating.
14. The apparatus of claim 13 , wherein the at least one processing device is configured to compare outputs generated by the retriever model and the generative model based on the validation set to annotations of the annotated document chunks, the annotations treated as ground truths.
15. A non-transitory computer readable medium containing instructions that when executed cause at least one processor to:
obtain an input query at a retriever model, the retriever model comprising a reconfigurable sequence of one or more rankers selected from among a plurality of rankers, each ranker configured to identify a specified number of information chunks relevant to the input query;
provide one or more of the information chunks from the retriever model to a generative model;
use the generative model to create a response to the input query, the response based on the one or more information chunks; and
determine the specified number of information chunks to be identified by each ranker in the reconfigurable sequence to tune the retriever model;
wherein the plurality of rankers comprises a bi-encoder, a cross-encoder, and a large language model (LLM)-ranker; and
wherein the instructions when executed cause the at least one processor to determine the specified number of information chunks to be identified by each ranker in the reconfigurable sequence using a grid search.
16. The non-transitory computer readable medium of claim 15 , wherein the retriever model is configured to process different fields of information in the information chunks differently.
17. The non-transitory computer readable medium of claim 15 , wherein the retriever model is configured to dynamically select a size of the one or more information chunks provided to the generative model.
18. The non-transitory computer readable medium of claim 15 , further containing instructions that when executed cause the at least one processor to provide the response to a user device associated with a user.
19. The non-transitory computer readable medium of claim 15 , wherein the instructions that when executed cause the at least one processor to determine the specified number of information chunks to be identified by each ranker in the reconfigurable sequence using the grid search comprise:
instructions that when executed cause the at least one processor to:
select values for the specified number of information chunks to be identified by each ranker in the reconfigurable sequence using a pre-defined grid;
use the retriever model and the generative model to process a validation set that includes annotated document chunks based on the selected values;
evaluate an accuracy of a system that includes the retriever model and the generative model based on the processing of the validation set; and
simultaneously tune the values for the specified number of information chunks to be identified by each ranker in the reconfigurable sequence based on results of the evaluating.
20. The non-transitory computer readable medium of claim 19 , wherein the instructions when executed cause the at least one processor to compare outputs generated by the retriever model and the generative model based on the validation set to annotations of the annotated document chunks, the annotations treated as ground truths.