Providing image context for information retrieval

1 . A system, comprising:

at least one processor; and

at least one memory coupled to the at least one processor and configured to store executable instructions executable by the at least one processor to cause the at least one processor to perform operations comprising:

extracting text from a document;

vectorizing the extracted text to obtain at least one vector;

extracting image data from the document;

storing, in an object storage, at least one image in the image data;

adding, to metadata of the at least one vector, a link to the stored at least one image; and

storing, in a vector database,

the at least one vector, and

the metadata of the at least one vector with the added link to the stored at least one image in the object storage,

wherein the operations further comprise, in response to a user query,

retrieving, from the vector database,

one or more vectors corresponding to the user query, and

one or more links to one or more images stored in the object storage, wherein the one or more links are included in metadata of the one or more vectors;

generating, using a Large Language Model (LLM), a response to the user query based on the one or more vectors; and

returning the generated response together with the one or more links to the one or more images stored in the object storage,

the retrieving the one or more vectors comprises:

vectorizing the user query to obtain at least one inquiry vector; and

searching the vector database for the one or more vectors that match the at least one inquiry vector, and

the operations further comprise:

extracting the one or more links to the one or more images from the metadata of the one or more vectors that match the at least one inquiry vector; and

removing duplicated links in the extracted one or more links, before returning together with the generated response.

2 . The system of claim 1 , wherein the operations further comprise:

performing artifact removal on the extracted image data to obtain the at least one image.

3 . The system of claim 2 , wherein the artifact removal comprises, for each candidate image in the image data,

discarding the candidate image, without including the candidate image in the at least one image to be stored in the object storage, in response to determining that

any of an image format, an image dimension, an image aspect ratio, mean saturation and a density of detected edges of the candidate image does not satisfy a corresponding predetermined criterion.

4 . The system of claim 3 , wherein the artifact removal comprises, for said each candidate image in the image data,

including the candidate image in the at least one image to be stored in the object storage, in response to determining that

all of the image format, the image dimension, the image aspect ratio, the mean saturation and the density of detected edges of the candidate image satisfy corresponding predetermined criteria, or

the image format, the image dimension and the image aspect ratio satisfy the corresponding predetermined criteria, and an optical character recognition (OCR) operation detects text in the candidate image.

5 . The system of claim 4 , wherein the artifact removal further comprises, for said each candidate image in the image data, at least one of

performing the OCR operation, in response to determining that

the image format, the image dimension and the image aspect ratio satisfy the corresponding predetermined criteria,

applying an image-blurring filter to the candidate image and then performing the OCR operation, in response to determining that

the image format, the image dimension and the image aspect ratio satisfy the corresponding predetermined criteria, or

converting the candidate image to a predetermined image mode before checking whether the image dimension satisfies the corresponding predetermined criterion, in response to determining that

the image format satisfies the corresponding predetermined criterion, and

an image mode of the candidate image is different from the predetermined image mode.

6 . The system of claim 5 , wherein the artifact removal comprises, for said each candidate image in the image data, at least one of applying a Gaussian blur algorithm as the image-blurring filter, performing an edge detection for detecting one or more edges in the candidate image, or using RGB as the predetermined image mode.

7 . The system of claim 1 , wherein

the vectorizing the extracted text comprises:

splitting the extracted text into a plurality of chunks; and

encoding the plurality of chunks correspondingly into a plurality of vectors, wherein the at least one vector comprises the plurality of vectors, and

a chunk location of each chunk among the plurality of chunks in the document is included in metadata of the corresponding vector among the plurality of vectors.

8 . The system of claim 7 , wherein

an image location of the at least one image in the document is included in metadata of the at least one image, and

the operations further comprise, for each vector among the plurality of vectors:

determining whether the chunk location in the metadata of the vector matches the image location in the metadata of the at least one image;

adding the image location and the link to the stored at least one image to the metadata of the vector, in response to determining that the chunk location matches the image location; and

not adding the image location and the link to the stored at least one image to the metadata of the vector, in response to determining that the chunk location does not match the image location.

9 . The system of claim 7 , wherein

the at least one image comprises a plurality of images,

each image among the plurality of images is stored in the object storage, and has metadata including:

a link to the stored image in the object storage, and

an image location of the image in the document, and the operations further comprise, for each vector among the plurality of vectors:

determining whether the chunk location in the metadata of the vector matches the image location in the metadata of any image among the plurality of images;

for any image among the plurality of images having the image location matching the chunk location, adding the image location and the link to the stored image to an array;

adding the array to the metadata of the vector; and

storing the vector having the metadata with the added array in the vector database.

10 . The system of claim 9 , wherein

the chunk location of each chunk among the plurality of chunks includes a page number of a page including the chunk in the document,

the image location of each image among the plurality of images includes a page number of a page including the image in the document, and

the chunk location of a vector among the plurality of vectors matches an image location of an image among the plurality of images where the corresponding chunk and the corresponding image are on a same page of the document.

11 . The system of claim 1 , wherein the operations further comprise, in response to the user query,

retrieving, using the one or more links, the one or more images from the object storage; and

providing, through a user interface, the generated response together with the retrieved one or more images.

12 . The system of claim 11 , wherein

the user interface comprises a web page or web browser through which

the user query is received from a user, and

the generated response together with the retrieved one or more images are provided in response to the user query.

13 . The system of claim 1 , wherein the generating the response to the user query comprises:

decoding the one or more vectors to obtain one or more chunks corresponding to the one or more vectors;

supplying the one or more chunks to the LLM; and

obtaining the generated response from the LLM in response to the supplied one or more chunks.

14 . A method, comprising:

extracting text from a document;

vectorizing the extracted text to obtain at least one vector;

extracting image data from the document;

storing, in an object storage, at least one image in the image data;

adding, to metadata of the at least one vector, a link to the stored at least one image; and

storing, in a vector database,

the at least one vector, and

the metadata of the at least one vector with the added link to the stored at least one image in the object storage,

wherein the method further comprises, in response to a user query,

retrieving, from the vector database,

one or more vectors corresponding to the user query, and

one or more links to one or more images stored in the object storage, wherein the one or more links are included in metadata of the one or more vectors;

generating, using a Large Language Model (LLM), a response to the user query based on the one or more vectors; and

returning the generated response together with the one or more links to the one or more images stored in the object storage,

the retrieving the one or more vectors comprises:

vectorizing the user query to obtain at least one inquiry vector; and

searching the vector database for the one or more vectors that match the at least one inquiry vector, and

the method further comprises:

extracting the one or more links to the one or more images from the metadata of the one or more vectors that match the at least one inquiry vector; and

removing duplicated links in the extracted one or more links, before returning together with the generated response.

15 . A non-transitory computer-readable medium having executable instructions recorded thereon that, in response to execution by at least one processor, cause performance of operations comprising:

extracting text from a document;

vectorizing the extracted text to obtain at least one vector;

extracting image data from the document;

storing, in an object storage, at least one image in the image data;

adding, to metadata of the at least one vector, a link to the stored at least one image; and

storing, in a vector database,

the at least one vector, and

the metadata of the at least one vector with the added link to the stored at least one image in the object storage,

wherein the operations further comprise, in response to a user query,

retrieving, from the vector database,

one or more vectors corresponding to the user query, and

one or more links to one or more images stored in the object storage, wherein the one or more links are included in metadata of the one or more vectors;

generating, using a Large Language Model (LLM), a response to the user query based on the one or more vectors; and

returning the generated response together with the one or more links to the one or more images stored in the object storage,

the retrieving the one or more vectors comprises:

vectorizing the user query to obtain at least one inquiry vector; and

searching the vector database for the one or more vectors that match the at least one inquiry vector, and

he operations further comprise:

extracting the one or more links to the one or more images from the metadata of the one or more vectors that match the at least one inquiry vector; and

removing duplicated links in the extracted one or more links, before returning together with the generated response.