Providing large language model embeddings using clusters and dictionaries

1 . A method for providing large language model (LLM) embeddings, the method comprising:

receiving an input query to a LLM service from a user;

determining if a first embedding of the input query is provided by a main embedding model;

based on a determination that the first embedding is provided by the main embedding model,

storing the input query and the first embedding as corresponding entries in an embeddings dictionary, wherein the first embedding being stored as a main embedding;

identifying an embeddings cluster similar to the input query, and

computing a main centroid embedding for the embeddings cluster based on the first embedding; and

based on a determination that the first embedding is not provided by the main embedding model,

generating a second embedding for the input query using a fallback embedding model,

searching the embeddings dictionary using the second embedding for a matching main embedding previously generated by the main embedding model,

based on a determination that matching main embedding is found in the embeddings dictionary, sending the matching main embedding to the LLM service, and

based on a determination that the matching main embedding is not found in the embeddings dictionary, identifying an embeddings cluster similar to the input query, and sending the main centroid embedding of the identified embeddings cluster to the LLM service.

2 . The method of claim 1 ,

wherein determining that the matching main embedding is found in the embeddings dictionary comprises: determining that a similarity between the second embedding and at least one entry in the embeddings dictionary is within a dictionary similarity threshold, and

wherein determining that the matching main embedding is not found in the embeddings dictionary comprises determining that the similarity between the second embedding and at least one entry in the embeddings dictionary is not within the dictionary similarity threshold.

3 . The method of claim 1 , wherein identifying an embeddings cluster similar to the input query further comprises:

providing a plurality of embeddings clusters, wherein each embeddings cluster comprises at least a secondary centroid embedding computed based on a plurality of second embeddings generated by the fallback embedding model, and a corresponding main centroid embedding computed based on a plurality of main embeddings generated by the main embedding model; and

selecting, out of the plurality of embeddings clusters, an embeddings cluster similar to the second embedding within a cluster similarity thresholds.

4 . The method of claim 3 , wherein a dictionary similarity threshold or the cluster similarity threshold is determined based on a similarity metric between distances of two or more embedding vectors.

5 . The method of claim 4 , wherein the similarity metric comprises one of: Jaccard similarity index and cosine similarity.

6 . The method of claim 3 , wherein the fallback embedding model is different from the main embedding model.

7 . The method of claim 3 , wherein the first embeddings and second embeddings have different vector dimensions.

8 . The method of claim 1 , wherein the main centroid embedding comprises an average or average dot product of two or more embedding vectors.

9 . The method of claim 3 , further comprising,

broaden a scope of search by adjusting the cluster similarity threshold if no embeddings cluster similar to the second embedding was found within the plurality of embeddings clusters.

10 . A system for providing large language model (LLM) embeddings, comprising:

at least one memory;

at least one hardware processor coupled with the at least one memory and configured, individually or in combination, to:

receive an input query to a LLM service from a user;

determine if a first embedding of the input query is provided by a main embedding model;

based on a determination that the first embedding is provided by the main embedding model,

store the input query and the first embedding as corresponding entries in an embeddings dictionary, wherein the first embedding being stored as a main embedding;

identify an embeddings cluster similar to the input query, and

compute a main centroid embedding for the embeddings cluster based on the first embedding; and

based on a determination that the first embedding is not provided by the main embedding model,

generate a second embedding for the input query using a fallback embedding model,

search the embeddings dictionary using the second embedding for a matching main embedding previously generated by the main embedding model,

based on a determination that matching main embedding is found in the embeddings dictionary, send the matching main embedding to the LLM service, and

based on a determination that the matching main embedding is not found in the embeddings dictionary, identify an embeddings cluster similar to the input query, and sending the main centroid embedding of the identified embeddings cluster to the LLM service.

11 . The system of claim 10 ,

wherein determining that the matching main embedding is found in the embeddings dictionary comprises: determining that a similarity between the second embedding and at least one entry in the embeddings dictionary is within a dictionary similarity threshold, and

wherein determining that the matching main embedding is not found in the embeddings dictionary comprises determining that the similarity between the second embedding and at least one entry in the embeddings dictionary is not within the dictionary similarity threshold.

12 . The system of claim 10 , wherein identifying an embeddings cluster similar to the input query further comprises:

providing a plurality of embeddings clusters, wherein each embeddings cluster comprises at least a secondary centroid embedding computed based on a plurality of second embeddings generated by the fallback embedding model, and a corresponding main centroid embedding computed based on a plurality of first embeddings generated by the main embedding model; and

selecting, out of the plurality of embeddings clusters, an embeddings cluster similar to the second embedding within a cluster similarity thresholds.

13 . The system of claim 12 , wherein a dictionary similarity threshold or the cluster similarity threshold is determined based on a similarity metric between distances of two or more embedding vectors.

14 . The system of claim 13 , wherein the similarity metric comprises one of: Jaccard similarity index and cosine similarity.

15 . The system of claim 12 , wherein the fallback embedding model is different from the main embedding model.

16 . The system of claim 12 , wherein the first embeddings and second embeddings have different vector dimensions.

17 . The system of claim 10 , wherein the main centroid embedding comprises an average or average dot product of two or more embedding vectors.

18 . The system of claim 12 , wherein at least one hardware processor coupled with the at least one memory and is further configured, individually or in combination, to:

broaden a scope of search by adjusting the cluster similarity threshold if no embeddings cluster similar to the second embedding was found within the plurality of embeddings clusters.

19 . A non-transitory computer readable medium storing thereon computer executable instructions for providing large language model (LLM) embeddings, including instructions for:

receiving an input query to a LLM service from a user;

determining if a first embedding of the input query is provided by a main embedding model;

based on a determination that the first embedding is provided by the main embedding model,

storing the input query and the first embedding as corresponding entries in an embeddings dictionary, wherein the first embedding being stored as a main embedding;

identifying an embeddings cluster similar to the input query, and

computing a main centroid embedding for the embeddings cluster based on the first embedding; and

based on a determination that the first embedding is not provided by the main embedding model,

generating a second embedding for the input query using a fallback embedding model,

searching the embeddings dictionary using the second embedding for a matching main embedding previously generated by the main embedding model,

based on a determination that matching main embedding is found in the embeddings dictionary, sending the matching main embedding to the LLM service, and

based on a determination that the matching main embedding is not found in the embeddings dictionary, identifying an embeddings cluster similar to the input query, and sending the main centroid embedding of the identified embeddings cluster to the LLM service.

20 . The non-transitory computer readable medium of claim 19 ,

wherein determining that the matching main embedding is found in the embeddings dictionary comprises: determining that a similarity between the second embedding and at least one entry in the embeddings dictionary is within a dictionary similarity threshold, and

wherein determining that the matching main embedding is not found in the embeddings dictionary comprises determining that the similarity between the second embedding and at least one entry in the embeddings dictionary is not within the dictionary similarity threshold.