Storing and searching sensitive data using embeddings
In an example embodiment, embeddings are used to securely search sensitive encrypted data stored in a database without the need to retrieve and decrypt all data in a volume. Specifically, a piece of text is embedded into an embedding, which is then stored in a vector database. Then the piece of text can be encrypted and stored in another database. A linkage between the embedding and the ciphertext can also be stored. When a search is performed, one or more keywords from the search can be embedded and the keyword embedding can be compared to the embeddings in the vector database to locate similar embeddings. The ciphertexts corresponding to those similar embeddings can then be retrieved from the other database and decrypted. The decrypted ciphertexts can then be searched based on the keywords to find exact matches.
1 . A system comprising:
at least one hardware processor; and
a computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
receiving a search query containing a first search condition;
passing the first search condition through an embedding machine learning model to create a search embedding, the search embedding being a vector of coordinates in a latent n-dimensional space;
locating, in a vector database storing a plurality of embeddings, one or more embeddings similar to the search embedding, each embedding comprising a vector of coordinates in a latent n-dimensional space;
grouping the one or more embeddings into a collection;
identifying, in a main database that stores encrypted data as ciphertext, one or more ciphertexts corresponding to the collection, based on one or more stored linkages between the one or more embeddings and the one or more ciphertexts;
decrypting the one or more identified ciphertexts into one or more plaintexts;
performing a search of the first search condition on the one or more plaintexts to locate one or more matching plaintexts; and
returning the one or more matching plaintexts in response to the search query.
2 . The system of claim 1 , wherein the locating comprises, for each embedding in the vector database, calculating a cosine correlation coefficient between the search embedding and the embedding and comparing the cosine correlation coefficient to a predetermined threshold.
3 . The system of claim 1 , wherein the embedding machine learning model is Large Language Model (LLM).
4 . The system of claim 1 , wherein the main database is a relational database.
5 . The system of claim 4 , wherein the main database is an in-memory database.
6 . The system of claim 4 , wherein the vector database is a non-relational database.
7 . The system of claim 1 , wherein the operations further comprise:
receiving a request to store a first piece of text in the main database;
encrypting the first piece of text into a first corresponding ciphertext;
storing the first corresponding ciphertext in the main database;
passing the first piece of text into the embedding machine learning model to create a first corresponding embedding;
generating a linkage between the first corresponding embedding and the first corresponding ciphertext; and
storing the first corresponding embedding and the linkage in the vector database.
8 . The system of claim 1 , wherein the search query further contains a second search condition separate from the first search condition by an operator, and wherein the operations further comprise:
passing second first search condition through the embedding machine learning model to create a second search embedding;
locating, in a vector database, one or more embeddings similar to the second search embedding;
modifying the collection based on the one or more embeddings similar to the second search embedding and the operator.
9 . The system of claim 8 , wherein the operator is an AND operator and the modifying comprises forming the collection using an intersection of the one or more embeddings similar to the search embedding and the one or more embeddings similar to the second search embedding.
10 . The system of claim 8 , wherein the operator is an OR operator and the modifying comprises forming the collection using a union of the one or more embeddings similar to the search embedding and the one or more embeddings similar to the second search embedding.
11 . A method comprising:
receiving a search query containing a first search condition;
passing the first search condition through an embedding machine learning model to create a search embedding, the search embedding being a vector of coordinates in a latent n-dimensional space;
locating, in a vector database storing a plurality of embeddings, one or more embeddings similar to the search embedding, each embedding comprising a vector of coordinates in a latent n-dimensional space;
grouping the one or more embeddings into a collection;
identifying, in a main database that stores encrypted data as ciphertext, one or more ciphertexts corresponding to the collection, based on one or more stored linkages between the one or more embeddings and the one or more ciphertexts;
decrypting the one or more identified ciphertexts into one or more plaintexts;
performing a search of the first search condition on the one or more plaintexts to locate one or more matching plaintexts; and
returning the one or more matching plaintexts in response to the search query.
12 . The method of claim 11 , wherein the locating comprises, for each embedding in the vector database, calculating a cosine correlation coefficient between the search embedding and the embedding and comparing the cosine correlation coefficient to a predetermined threshold.
13 . The method of claim 11 , wherein the embedding machine learning model is Large Language Model (LLM).
14 . The method of claim 11 , wherein the main database is a relational database.
15 . The method of claim 14 , wherein the main database is an in-memory database.
16 . The method of claim 14 , wherein the vector database is a non-relational database.
17 . The method of claim 11 , further comprising:
receiving a request to store a first piece of text in the main database;
encrypting the first piece of text into a first corresponding ciphertext;
storing the first corresponding ciphertext in the main database;
passing the first piece of text into the embedding machine learning model to create a first corresponding embedding;
generating a linkage between the first corresponding embedding and the first corresponding ciphertext; and
storing the first corresponding embedding and the linkage in the vector database.
18 . The method of claim 11 , wherein the search query further contains a second search condition separate from the first search condition by an operator, and wherein the method further comprises:
passing second first search condition through the embedding machine learning model to create a second search embedding;
locating, in a vector database, one or more embeddings similar to the second search embedding;
modifying the collection based on the one or more embeddings similar to the second search embedding and the operator.
19 . The method of claim 18 , wherein the operator is an AND operator and the modifying comprises forming the collection using an intersection of the one or more embeddings similar to the search embedding and the one or more embeddings similar to the second search embedding.
20 . A non-transitory machine-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a search query containing a first search condition;
passing the first search condition through an embedding machine learning model to create a search embedding, the search embedding being a vector of coordinates in a latent n-dimensional space;
locating, in a vector database storing a plurality of embeddings, one or more embeddings similar to the search embedding, each embedding comprising a vector of coordinates in a latent n-dimensional space;
grouping the one or more embeddings into a collection;
identifying, in a main database that stores encrypted data as ciphertext, one or more ciphertexts corresponding to the collection, based on one or more stored linkages between the one or more embeddings and the one or more ciphertexts;
decrypting the one or more identified ciphertexts into one or more plaintexts;
performing a search of the first search condition on the one or more plaintexts to locate one or more matching plaintexts; and
returning the one or more matching plaintexts in response to the search query.