Vector generation and indexing system

1 . A computer-implemented method comprising:

generating high-dimensional vector embeddings for sets of data to be stored in each of a respective table of a plurality of tables;

storing the high-dimensional vector embeddings for each set of data of the sets of data in the respective table of the plurality of tables;

storing, for each table, a model type and version corresponding to a model used to generate the high-dimensional vector embeddings stored in each respective table of the plurality of tables;

receiving a query and determining a subset of tables of the plurality of tables that are relevant to the query;

determining a stored model type and version associated with each table of the subset of tables relevant to the query, wherein the stored model type and version comprises at least a first model type and a first version and a second model type and a second version;

generating a first high-dimensional vector embedding for the query using at least the first model type and first version and a second high-dimensional vector embedding for the query using the second model type and second version;

generating a set of similar data based on at least the first high-dimensional vector embedding and the second high-dimensional vector embedding for the query and the high-dimensional vector embeddings for the sets of data stored in the subset of tables; and

generating, in real time or near real time from receipt of the query, a response to the query based on the set of similar data.

2 . The computer-implemented method of claim 1 , wherein after generating high-dimensional vector embeddings for sets of data to be stored in each of a respective table of the plurality of tables, the method comprises:

generating a vector index for each table of the plurality of tables.

3 . The computer-implemented method of claim 2 , wherein generating the set of similar data based on at least the first high-dimensional vector embedding and the second high-dimensional vector embedding for the query and the high-dimensional vector embeddings for the sets of data stored in the subset of tables comprises using a function that can compare two vectors for similarity and utilize the vector index to compare the two vectors.

4 . The computer-implemented method of claim 1 , wherein at least two different model types are used to generate high-dimensional vector embeddings.

5 . The computer-implemented method of claim 1 , wherein at least two different versions of a model type are used to generate high-dimensional vector embeddings.

6 . The computer-implemented method of claim 1 , wherein generating high-dimensional vector embeddings for sets of data comprises converting each piece of data of each set of data to a vector space with a given number of dimensions.

7 . The computer-implemented method of claim 1 , wherein the set of similar data is determined based on at least on a comparison of the first high-dimensional vector embedding of the query and the high-dimensional vector embeddings for the sets of data stored in a first table and a comparison of the second high-dimensional vector embedding of the query and high-dimensional vector embeddings for the sets of data stored in a second table.

8 . The computer-implemented method of claim 7 , wherein the comparison is made using a function that can compare two vectors for similarity.

9 . The computer-implemented method of claim 1 , wherein the high-dimensional vector embeddings for a set of data stored in a first table were generated using a first model type and a first version, and further comprising:

re-generating high-dimensional vector embeddings for the set of data stored in the first table using the first model type and a second version;

storing the re-generated high-dimensional vector embeddings in the first table; and

updating, for the first table, the version corresponding to the first model type used to re-generate the high-dimensional vector embeddings for the set of data stored in the first table.

10 . The computer-implemented method of claim 9 , further comprising:

receiving a second query corresponding to the first table;

determining the first model type and second version associated with the first table;

generating a high-dimensional vector embedding for the second query using the first model type and second version associated with the first table;

determining a second set of similar data in the first table based on the high-dimensional vector embeddings of the second query and the high-dimensional vector embeddings for the sets of data stored in the first table; and

generating a response to the second query based on the second set of similar data.

11 . The computer-implemented method of claim 1 , wherein data comprises text, an image, audio, or video.

12 . A system comprising:

a memory that stores instructions; and

one or more processors configured by the instructions to perform operations comprising:

generating high-dimensional vector embeddings for sets of data to be stored in each of a respective table of a plurality of tables;

storing the high-dimensional vector embeddings for each set of data of the sets of data in the respective table of the plurality of tables;

storing, for each table, a model type and version corresponding to a model used to generate the high-dimensional vector embeddings stored in each respective table of the plurality of tables;

receiving a query and determining a subset of tables of the plurality of tables that are relevant to the query;

determining a stored model type and version associated with each table of the subset of tables relevant to the query, wherein the stored model type and version comprises at least a first model type and a first version and a second model type and a second version;

generating a first high-dimensional vector embedding for the query using at least the first model type and first version and a second high-dimensional vector embedding for the query using the second model type and second version;

generating a set of similar data based on at least the first high-dimensional vector embedding and the second high-dimensional vector embedding for the query and the high-dimensional vector embeddings for the sets of data stored in the subset of tables; and

generating, in real time or near real time from receipt of the query, a response to the query based on the set of similar data.

13 . The system of claim 12 , wherein after generating high-dimensional vector embeddings for sets of data to be stored in each of a respective table of the plurality of tables, the operations comprise:

generating a vector index for each table of the plurality of tables.

14 . The system of claim 12 , wherein at least two different model types are used to generate high-dimensional vector embeddings or wherein at least two different versions of a model type are used to generate high-dimensional vector embeddings.

15 . The system of claim 12 , wherein generating high-dimensional vector embeddings for sets of data comprises converting each piece of data of each set of data to a vector space with a given number of dimensions.

16 . The system of claim 12 , wherein the set of similar data is determined based on at least on a comparison of the first high-dimensional vector embeddings of the query and the high-dimensional vector embeddings for the sets of data stored in a first table and a comparison of the second high-dimensional vector embeddings of the query and vector embeddings for the sets of data stored in a second table.

17 . The system of claim 16 , wherein the comparison is made using a function that can compare two vectors for similarity.

18 . The system of claim 12 , wherein the high-dimensional vector embeddings for a set of data stored in a first table were generated using a first model type and a first version, and the operations further comprising:

re-generating high-dimensional vector embeddings for the set of data stored in the first table using the first model type and a second version;

storing the re-generated high-dimensional vector embeddings in the first table; and

updating, for the first table, the version corresponding to the first model type used to re-generate the high-dimensional vector embeddings for the set of data stored in the first table.

19 . The system of claim 18 , the operations further comprising:

receiving a second query corresponding to the first table;

determining the first model type and second version associated with the first table;

generating a high-dimensional vector embedding for the second query using the first model type and second version associated with the first table;

determining a second set of similar data in the first table based on the high-dimensional vector embedding of the second query and the high-dimensional vector embeddings for the sets of data stored in the first table; and

generating a response to the second query based on the second set of similar data.

20 . A non-transitory computer-readable medium comprising instructions stored thereon that are executable by at least one processor to cause a computing device to perform operations comprising:

generating high-dimensional vector embeddings for sets of data to be stored in each of a respective table of a plurality of tables;

storing the high-dimensional vector embeddings for each set of data of the sets of data in the respective table of the plurality of tables;

storing, for each table, a model type and version corresponding to a model used to generate the high-dimensional vector embeddings stored in each respective table of the plurality of tables;

receiving a query and determining a subset of tables of the plurality of tables that are relevant to the query;

determining a stored model type and version associated with each table of the subset of tables relevant to the query, wherein the stored model type and version comprises at least a first model type and a first version and a second model type and a second version;

generating a first high-dimensional vector embedding for the query using at least the first model type and first version and a second high-dimensional vector embedding for the query using the second model type and second version;

generating a set of similar data based on at least the first high-dimensional vector embedding and the second high-dimensional vector embedding for the query and the high-dimensional vector embeddings for the sets of data stored in the subset of tables; and

generating, in real time or near real time from receipt of the query, a response to the query based on the set of similar data.