Tuning a retrieval-augmented generation pipeline based on context and domain

1 . A method comprising:

acquiring, by a processing system including at least one processor, data from at least one data source of an enterprise;

detecting, by the processing system, a plurality of characteristics of the data;

selecting, by the processing system based on the plurality of characteristics, a partitioning technique for partitioning the data;

partitioning, by the processing system using the partitioning technique, the data to produce a plurality of data chunks;

generating, by the processing system, a plurality of vector embeddings of the plurality of data chunks;

selecting, by the processing system, metadata with which to augment the plurality of vector embeddings;

augmenting, by the processing system, the plurality of vector embeddings with the metadata;

applying, by the processing system, an indexing algorithm to the plurality of vector embeddings to facilitate a storage of the plurality of vector embeddings; and

wherein the method trains the processing system to perform operations related to a retrieval-augmented generation using the data by:

receiving, by the processing system, a user query for information stored in the at least one data source;

adapting, by the processing system, the user query to align with a context of the enterprise to produce an adapted user query;

applying, by the processing system, a vector compression technique to the adapted user query to minimize a latency in a retrieval of initial search results by a hybrid search executing the adapted user query against the at least one data source;

performing, by the processing system, the hybrid search of the at least one data source, based on the adapted user query, where the hybrid search combines a keyword-based search technique and a vector-based search technique;

adjusting, by the processing system based on the context of the enterprise, a number of the initial search results of the hybrid search to input into a re-ranking model; and

re-ranking, by the processing system, the initial search results using the re-ranking model to produce refined search results.

2 . The method of claim 1 , wherein the at least one data source is one of a plurality of data sources associated with the enterprise, and wherein at least two data sources of the plurality of data sources store different types of data.

3 . The method of claim 1 , wherein each data chunk of the plurality of data chunks comprises a subset of the data that is less than an entirety of the data.

4 . The method of claim 1 , wherein the partitioning technique is at least one of: a fixed size chunking technique, a content aware chunking technique, a recursive chunking technique, a specialized chunking technique, or a semantic chunking technique.

5 . The method of claim 1 , wherein the partitioning technique is selected to produce a desired chunk size and a desired degree of overlap among the plurality of data chunks.

6 . The method of claim 5 , wherein the desired chunk size and the desired degree of overlap are tuned to preserve contextual information in the plurality of data chunks.

7 . The method of claim 5 , wherein the at least one data source comprises a first data source and a second data source, and the partitioning technique selected to partition a portion of the data acquired from the first data source is different from the partitioning technique selected to partition a portion of the data acquired from the second data source.

8 . The method of claim 1 , wherein the plurality of vector embeddings is generated using an embedding model that has been trained on a terminology that is unique to the enterprise.

9 . The method of claim 8 , wherein the terminology includes terms that have meanings to the enterprise that are different from commonly understood meanings of the terms.

10 . The method of claim 9 , wherein the enterprise comprises a telecommunications network service provider, and the terms relate to at least one of: a telecommunications standard, a technology, or a deployment scenario.

11 . The method of claim 1 , wherein the enterprise comprises a telecommunications network service provider, and the metadata includes data related to at least one of: a configuration of an associated telecommunications network, a topology of an associated telecommunications network, or an associated service type which varies with a generation of technology the service type belongs to.

12 . The method of claim 1 , wherein the indexing algorithm creates different indices for different types of data.

13 . The method of claim 1 , wherein the indexing algorithm comprises a multi-indexing algorithm.

14 . The method of claim 1 , wherein the adapting comprises rephrasing the user query to better match the context of the enterprise.

15 . The method of claim 1 , wherein the vector compression technique comprises at least one of: a discrete cosine transform, a wavelet transform, a Huffman coding, a scalar quantization, a k-means clustering, a LZ77, wavelet decomposition, or a classified vector quantization.

16 . The method of claim 1 , wherein the refined search results are better aligned with the context of the enterprise than the initial search results.

17 . A non-transitory computer-readable medium storing instructions which, when executed by a processing system including at least one processor, cause the processing system to perform operations, the operations comprising:

acquiring data from at least one data source of an enterprise;

detecting a plurality of characteristics of the data;

selecting, based on the plurality of characteristics, a partitioning technique for partitioning the data;

partitioning, using the partitioning technique, the data to produce a plurality of data chunks;

generating a plurality of vector embeddings of the plurality of data chunks;

selecting metadata with which to augment the plurality of vector embeddings;

augmenting the plurality of vector embeddings with the metadata;

applying an indexing algorithm to the plurality of vector embeddings to facilitate a storage of the plurality of vector embeddings; and

wherein the operations further train the processing system to perform operations related to a retrieval-augmented generation using the data by:

receiving a user query for information stored in the at least one data source;

adapting the user query to align with a context of the enterprise to produce an adapted user query;

applying a vector compression technique to the adapted user query to minimize a latency in a retrieval of initial search results by a hybrid search executing the adapted user query against the at least one data source;

performing the hybrid search of the at least one data source, based on the adapted user query, where the hybrid search combines a keyword-based search technique and a vector-based search technique;

adjusting, based on the context of the enterprise, a number of the initial search results of the hybrid search to input into a re-ranking model; and

re-ranking the initial search results using the re-ranking model to produce refined search results.

18 . A device comprising:

a processing system including at least one processor; and

a non-transitory computer-readable medium storing instructions which, when executed by the processing system, cause the processing system to perform operations, the operations comprising:

acquiring data from at least one data source of an enterprise;

detecting a plurality of characteristics of the data;

selecting, based on the plurality of characteristics, a partitioning technique for partitioning the data;

partitioning, using the partitioning technique, the data to produce a plurality of data chunks;

generating a plurality of vector embeddings of the plurality of data chunks;

selecting metadata with which to augment the plurality of vector embeddings;

augmenting the plurality of vector embeddings with the metadata;

applying an indexing algorithm to the plurality of vector embeddings to facilitate a storage of the plurality of vector embeddings; and

wherein the operations further train the processing system to perform operations related to a retrieval-augmented generation using the data by:

receiving a user query for information stored in the at least one data source;

adapting the user query to align with a context of the enterprise to produce an adapted user query;

applying a vector compression technique to the adapted user query to minimize a latency in a retrieval of initial search results by a hybrid search executing the adapted user query against the at least one data source;

performing the hybrid search of the at least one data source, based on the adapted user query, where the hybrid search combines a keyword-based search technique and a vector-based search technique;

adjusting, based on the context of the enterprise, a number of the initial search results of the hybrid search to input into a re-ranking model; and

re-ranking the initial search results using the re-ranking model to produce refined search results.

19 . The method of claim 12 , wherein the different indices enable the plurality of vector embeddings to be stored in a manner that improves an efficiency with which the hybrid search is able to access the plurality of vector embeddings.

20 . The method of claim 1 , wherein the combining the keyword-based technique and the vector-based search technique balances meanings of enterprise-specific and industry-specific terms in a manner that improves a relevance of the initial search results to the context of the enterprise.