Reference file management for artificial intelligence models
A Data Storage Device (DSD) includes a first memory storing reference files used to derive vector embeddings in a vector database. A query vector embedding is received from a host and one or more vector embeddings similar to the query vector embedding are identified in the vector database. One or more reference files from which the one or more vector embeddings were derived are identified and stored in a second memory for faster access. In another aspect, a query vector embedding is received by a host that identifies one or more vector embeddings that are similar to the query vector embedding and retrieves the one or more vector embeddings from a DSD to provide to an Artificial Intelligence (AI) model. One or more reference files are identified from which the one or more vector embeddings were derived and are prefetched from the DSD for storage in a host memory.
1. A Data Storage Device (DSD), comprising:
a first memory configured to store a plurality of reference files that have been used to derive a plurality of vector embeddings in a vector database stored in the DSD; and
circuitry configured to:
receive a query vector embedding from a host;
perform an Approximate Nearest Neighbor (ANN) search of the vector database to identify one or more vector embeddings that are close to the query vector embedding;
send the one or more vector embeddings to the host;
identify one or more reference files from which the one or more vector embeddings were derived using vector metadata for the one or more vector embeddings; and
store the one or more reference files in a second memory of the host or of the DSD, the second memory configured to provide access to data faster than the first memory.
2. The DSD of claim 1 , wherein the circuitry is further configured to create a zone in the second memory for caching reference files from the plurality of reference files stored in the first memory.
3. The DSD of claim 1 , wherein the circuitry is further configured to prioritize storage of the one or more reference files in the second memory over at least one other operation performed by the DSD.
4. The DSD of claim 1 , wherein the circuitry is further configured to:
associate the one or more reference files stored in the second memory with the respective one or more vector embeddings derived from the one or more reference files;
receive a request from the host for a reference file corresponding to the one or more vector embeddings;
identify the requested reference file stored in the second memory using the association; and
send the requested reference file stored in the second memory to the host.
5. The DSD of claim 1 , wherein the circuitry is further configured to perform the ANN search by comparing the query vector embedding with a plurality of candidate vector embeddings in the vector database to identify the one or more vector embeddings that are closest to the query vector embedding.
6. The DSD of claim 5 , wherein the circuitry is further configured to perform the ANN search of the vector database by at least one of determining a cosine of an angle between candidate vector embeddings and the query vector embedding, determining a Euclidian distance between candidate vector embeddings and the query vector embedding and determining a dot product between candidate vector embeddings and the query vector embedding to identify the one or more vector embeddings.
7. The DSD of claim 1 , wherein the circuitry is further configured to use the vector metadata as part of a filtering operation in a search for the one or more vector embeddings.
8. The DSD of claim 1 , wherein the second memory is further configured to store an index that includes vector metadata for the plurality of vector embeddings in the vector database, and wherein the circuitry is further configured to identify the vector metadata for the one or more vector embeddings using the index.
9. A method performed by a host for an Artificial Intelligence (AI) model, the method comprising:
receiving a query vector embedding;
performing an Approximate Nearest Neighbor (ANN) search of a vector database to identify one or more vector embeddings in the vector database that are similar to the query vector embedding;
retrieving the one or more vector embeddings from the vector database;
providing the one or more vector embeddings to the AI model;
identifying one or more reference files from which the one or more vector embeddings were derived using vector metadata for the one or more vector embeddings;
prefetching the one or more reference files from a Data Storage Device (DSD); and
storing the prefetched one or more reference files in at least one memory of the host.
10. The method of claim 9 , further comprising creating a zone in the at least one memory of the host for caching reference files from a plurality of reference files stored in the DSD.
11. The method of claim 9 , further comprising prioritizing the prefetching of the one or more reference files from the DSD over at least one other operation performed by at least one of the host and the DSD.
12. The method of claim 9 , further comprising:
associating the one or more reference files stored in the at least one memory with the respective one or more vector embeddings derived from the one or more reference files;
receiving a request from an application for a reference file corresponding to the one or more vector embeddings;
identifying the requested reference file stored in the at least one memory using the association; and
sending the requested reference file stored in the at least one memory to the application.
13. The method of claim 9 , further comprising performing the ANN search by comparing the query vector embedding with a plurality of candidate vector embeddings in the vector database to identify the one or more vector embeddings that are closest to the query vector embedding.
14. The method of claim 13 , further comprising performing the ANN search of the vector database by at least one of determining a cosine of an angle between candidate vector embeddings and the query vector embedding, determining a Euclidian distance between candidate vector embeddings and the query vector embedding and determining a dot product between candidate vector embeddings and the query vector embedding.
15. The method of claim 9 , further comprising using the vector metadata as part of a filtering operation in a search for the one or more vector embeddings.
16. The method of claim 9 , wherein the at least one memory of the host further stores an index that includes vector metadata for a plurality of vector embeddings in the vector database, and wherein the method further comprises identifying the vector metadata for the one or more vector embeddings using the index.
17. A Data Storage Device (DSD), comprising:
a first memory configured to store a plurality of reference files that have been used to derive a plurality of vector embeddings in a vector database stored in the DSD; and
means for:
receiving a query vector embedding from a host;
performing an Approximate Nearest Neighbor (ANN) search of the vector database to identify one or more vector embeddings that are close to the query vector embedding;
sending the one or more vector embeddings to the host;
identifying one or more reference files from which the one or more vector embeddings were derived using vector metadata for the one or more vector embeddings; and
storing the one or more reference files in a second memory of the host or of the DSD, the second memory configured to provide access to data faster than the first memory.
18. The DSD of claim 17 , further comprising means for prioritizing the storage of the one or more reference files in the second memory over at least one other operation performed by the DSD.
19. The DSD of claim 17 , further comprising means for:
associating the one or more reference files stored in the second memory with the respective one or more vector embeddings derived from the one or more reference files;
receiving a request from the host for a reference file corresponding to the one or more vector embeddings;
identifying the requested reference file stored in the second memory using the association; and
sending the requested reference file stored in the second memory to the host.
20. The DSD of claim 17 , wherein the second memory is further configured to store an index that includes vector metadata for the plurality of vector embeddings in the vector database, and wherein the DSD further comprises means for identifying the vector metadata for the one or more vector embeddings using the index.