Automatic pipeline for embedding vector generation

1 . A method, comprising:

receiving a request from a user, the request specifying a task for generating one or more embedding vectors to represent a target dataset using a target embedding model selected by the user from among a plurality of embedding models;

applying, based on the request, an automatic pipeline to the target dataset, wherein applying the automatic pipeline comprises:

pre-processing the target dataset to generate one or more batches of the target dataset, each batch of the target dataset comprising a plurality of data chunks in the target dataset;

assigning, based in part on the target embedding model selected by the user, a dynamically adjustable number of computing resources to each batch of the target dataset to generate embedding vectors for the batch;

applying, using respective dynamically adjustable number of computing resources, the target embedding model to each batch of the target dataset; and

generating the one or more embedding vectors, each representing a data chunk of the target dataset in a latent space, wherein each embedding vector is incrementally updated based on an update to a corresponding data chunk represented by the embedding vector; and

storing, in a vector database, the generated one or more embedding vectors for representing the target dataset in the latent space.

2 . The method of claim 1 , wherein applying the automatic pipeline further comprises:

detecting an error characteristic during generating the one or more embedding vectors;

determining that the error characteristic is related to the computing resources; and

automatically adjusting for a given batch, based on the error characteristic, the dynamically adjustable number of computing resources.

3 . The method of claim 1 , wherein generating the one or more embedding vectors further comprises:

detecting a change to a first dataset that is represented by a first set of embedding vectors in the vector database;

determining that the change to the first dataset is related to a first data chunk in the first dataset;

updating a first embedding vector representing the first data chunk with the detected change; and

storing the updated first embedding vector in the vector database.

4 . The method of claim 1 , comprising:

receiving, from a client device, a query;

identifying, based on the query, one or more datasets using a plurality of embedding vectors in the vector database;

generating a prompt for input to a machine-learned language model, the prompt specifying at least the query, the one or more identified datasets, and a request to generate a response to the query using the one or more identified datasets as contextual information of the query;

providing the prompt to a model serving system for execution by the machine-learned language model;

receiving, from the model serving system, an output generated by executing the machine-learned language model on the prompt, the output comprising the requested response to the query; and

providing for display a query response including the received output.

5 . The method of claim 4 , wherein identifying, based on the query, the one or more datasets using the plurality of embedding vectors further comprises:

applying the target embedding model to the query; and

generating a user query embedding vector that represents the query in the latent space.

6 . The method of claim 1 , further comprising:

periodically updating the one or more embedding vectors; and

storing the updated embedding vectors in the vector database.

7 . The method of claim 1 , wherein the target dataset comprises information in a format of text, audio, video, or image.

8 . A non-transitory computer readable storage medium comprising stored instructions that when executed by one or more processors of one or more computing devices, cause the one or more computing devices to:

receive a request from a user, the request specifying a task for generating one or more embedding vectors to represent a target dataset using a target embedding model selected by the user from among a plurality of embedding models;

apply, based on the request, an automatic pipeline to the target dataset, wherein the instructions to apply the automatic pipeline, cause the one or more computing devices to:

pre-process the target dataset to generate one or more batches of the target dataset, each batch of the target dataset comprising a plurality of data chunks in the target dataset;

assign, based in part on the target embedding model selected by the user, a dynamically adjustable number of computing resources to each batch of the target dataset to generate embedding vectors for the batch;

apply, using respective dynamically adjustable number of computing resources, the target embedding model to each batch of the target dataset; and

generate the one or more embedding vectors, each representing a data chunk of the target dataset in a latent space, wherein each embedding vector is incrementally updated based on an update to a corresponding data chunk represented by the embedding vector; and

store, in a vector database, the generated one or more embedding vectors for representing the target dataset in the latent space.

9 . The non-transitory computer readable storage medium of claim 8 , the instructions to apply the automatic pipeline, cause the one or more computing devices to:

detect an error characteristic during generating the one or more embedding vectors;

determine that the error characteristic is related to the computing resources; and

automatically adjust for a given batch, based on the error characteristic, the dynamically adjustable number of computing resources.

10 . The non-transitory computer readable storage medium of claim 8 , wherein the instructions to generate the one or more embedding vectors cause the one or more computing devices to:

detect a change to a first dataset that is represented by a first set of embedding vectors in the vector database;

determine that the change to the first dataset is related to a first data chunk in the first dataset;

update a first embedding vector representing the first data chunk with the detected change; and

store the updated first embedding vector in the vector database.

11 . The non-transitory computer readable storage medium of claim 8 , wherein the instructions further cause the one or more computing devices to:

receive, from a client device, a query;

identify, based on the query, one or more datasets using a plurality of embedding vectors in the vector database;

generate a prompt for input to a machine-learned language model, the prompt specifying at least the query, the one or more identified datasets, and a request to generate a response to the query using the one or more identified datasets as contextual information of the query;

provide the prompt to a model serving system for execution by the machine-learned language model;

receive, from the model serving system, an output generated by executing the machine-learned language model on the prompt, the output comprising the requested response to the query; and

provide for display a query response including the received output.

12 . The non-transitory computer readable storage medium of claim 11 , wherein the instructions to identify the one or more datasets using the plurality of embedding vectors cause the one or more computing devices to:

apply the target embedding model to the query; and

generate a user query embedding vector that represents the query in the latent space.

13 . The non-transitory computer readable storage medium of claim 8 , wherein the instructions further cause the one or more computing devices to:

periodically update the one or more embedding vectors; and

store the updated embedding vectors in the vector database.

14 . The non-transitory computer readable storage medium of claim 8 , wherein the target dataset comprises information in a format of text, audio, video, or image.

15 . A system comprising:

one or more computer processors; and

one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the system to:

receive a request from a user, the request specifying a task for generating one or more embedding vectors to represent a target dataset using a target embedding model selected by the user from among a plurality of embedding models;

apply, based on the request, an automatic pipeline to the target dataset, wherein the instructions to apply the automatic pipeline, cause the system to:

pre-process the target dataset to generate one or more batches of the target dataset, each batch of the target dataset comprising a plurality of data chunks in the target dataset;

assign, based in part on the target embedding model selected by the user, a dynamically adjustable number of computing resources to each batch of the target dataset to generate embedding vectors for the batch;

apply, using respective dynamically adjustable number of computing resources, the target embedding model to each batch of the target dataset; and

generate the one or more embedding vectors, each representing a data chunk of the target dataset in a latent space, wherein each embedding vector is incrementally updated based on an update to a corresponding data chunk represented by the embedding vector; and

store, in a vector database, the generated one or more embedding vectors for representing the target dataset in the latent space.

16 . The system of claim 15 , wherein the instructions to apply the automatic pipeline further cause the system to:

detect an error characteristic during generating the one or more embedding vectors;

determine that the error characteristic is related to the computing resources; and

automatically adjust for a given batch, based on the error characteristic, the dynamically adjustable number of computing resources.

17 . The system of claim 15 , wherein the instructions further cause the system to:

receive, from a client device, a query;

identify, based on the query, one or more datasets using a plurality of embedding vectors in the vector database;

generate a prompt for input to a machine-learned language model, the prompt specifying at least the query, the one or more identified datasets, and a request to generate a response to the query using the one or more identified datasets as contextual information of the query;

provide the prompt to a model serving system for execution by the machine-learned language model;

receive, from the model serving system, an output generated by executing the machine-learned language model on the prompt, the output comprising the requested response to the query; and

provide for display a query response including the received output.

18 . The system of claim 17 , wherein the instructions to identify the one or more datasets using the plurality of embedding vectors further cause the system to:

apply the target embedding model to the query; and

generate a user query embedding vector that represents the query in the latent space.

19 . The system of claim 15 , wherein the instructions further cause the system to:

periodically update the one or more embedding vectors; and

store the updated embedding vectors in the vector database.