Storing entries in and retrieving information from an embedding object memory
Methods, systems, and media for storing entries in and/or retrieving information from an embedding object memory are provided. In some examples, a content item is received that has content data. The content data associated with the content item may be provided to one or more semantic embedding models that generate semantic embeddings. From one or more of the semantic embedding models, one or semantic embeddings may be received. The one or more semantic embeddings may then be inserted into the embedding object memory. The semantic embeddings may be associated with respective indications corresponding to a reference to source data associated with the semantic embeddings. Further, the insertion may trigger a spatial storage operation to store a vector representation of the one or more semantic embeddings. A plurality of collections of stored embeddings may be received from the embedding object memory, based on a provided input, to determine an action.
1. A method for storing an entry in an embedding object memory, the method comprising:
receiving a content item, the content item having one or more content data;
providing one of the content data associated with the content item to one or more semantic embedding models, wherein the one or more semantic embedding models generate one or more semantic embeddings;
receiving, from one or more of the semantic embedding models, one or more semantic embeddings, wherein a collection of semantic embeddings is associated with a first semantic embedding model of the one or more semantic embedding models, wherein the collection of semantic embeddings comprises a first semantic embedding generated by the first semantic embedding model for at least one content data from the respective content item, wherein the one or more semantic embedding models comprise a version, and wherein each of the semantic embeddings generated by each of the respective one or more semantic embedding models comprise metadata corresponding to the version;
inserting the one or more semantic embeddings into the embedding object memory, wherein the embedding object memory stores one or more semantic embeddings from the collection of semantic embeddings, wherein the one or more semantic embeddings are associated with a respective indication corresponding to a reference to source data associated with the one or more semantic embeddings, and wherein the insertion triggers a spatial storage operation to store a vector representation of the one or more semantic embeddings;
providing an updated semantic embedding model to replace at least one of the semantic embedding models, the updated semantic embedding model comprising an updated version that is different than the version of the at least one of the semantic embedding models;
receiving, from the updated semantic embedding model, an updated one or more semantic embeddings corresponding to the one or more semantic embeddings generated by the at least one of the semantic embedding models, wherein the updated one or more semantic embeddings are generated based on the one of the content data used to generate the one or more semantic embeddings;
inserting the updated semantic embeddings in the embedding object memory with metadata corresponding to the updated version; and
providing the embedding object memory.
2. The method of claim 1 , wherein the vector representation is stored in at least one of an approximate nearest neighbor (ANN) tree, a k-d tree, or a multidimensional tree.
3. The method of claim 1 , wherein the semantic embedding models comprise a generative large language model (LLM).
4. The method of claim 1 , wherein the content data are a plurality of content data that are each a respective one of audio content data, visual content data, gaze content data, weather content data, news content data, calendar content data, email content data, or location content data.
5. The method of claim 1 , wherein the embedding object memory is stored at a location that is different than the location of the source data.
6. The method of claim 1 , further comprising:
deleting the one or more semantic embeddings corresponding to the updated semantic embeddings, the one or more semantic embeddings having metadata corresponding to a version that is different than the updated version.
7. A method for retrieving information from an embedding object memory, the method comprising:
receiving an input embedding, wherein the input embedding is generated by a machine-learning model;
retrieving a plurality of collections of stored semantic embeddings, from the embedding object memory, based on the input embedding, wherein the plurality of collections of stored semantic embeddings each correspond to respective content data;
retrieving a subset of semantic embeddings from at least one of the plurality of collections of stored semantic embeddings based on a similarity to the input embedding, wherein the subset of semantic embeddings includes a first semantic embedding stored with metadata corresponding to a first version of a semantic embedding model used to generate the first semantic embedding based on a content item, and wherein the subset of semantic embeddings further includes a second semantic embedding stored with metadata corresponding to a second version of the semantic embedding model used to generate the second semantic embedding based on the content item;
determining, based on the subset of semantic embeddings and the input embedding, an action; and
providing the action as an output.
8. The method of claim 7 , wherein each semantic embedding of the subset of semantic embeddings is associated with source data corresponding to the respective content data, wherein the determining an action comprises:
locating the source data; and
determining the action based on the input embedding and the source data.
9. The method of claim 8 , wherein the source data includes one or more of audio files, text files, or image files.
10. The method of claim 7 , wherein the retrieving a subset of embeddings comprises:
determining a respective similarity between the input embedding and each embedding of the plurality of collections of stored semantic embeddings;
determining an ordered ranking of the one or more similarities or that one or more of the similarities are less than a predetermined threshold; and
retrieving the subset of semantic embeddings with similarities to the input embedding that are less than the predetermined threshold or based on the ordered ranking, thereby retrieving subsets of semantic embeddings from at least one of the collections of semantic embeddings that are determined to be related to the input embedding.
11. The method of claim 10 , wherein the similarities are distances.
12. The method of claim 7 , further comprising, prior to receiving the input embedding:
receiving user-input; and
generating the input embedding based on the user-input.
13. The method of claim 7 , further comprising:
adapting a computing device to perform the action.
14. The method of claim 7 , wherein the machine-learning model is a generative multimodal machine-learning model.
15. A method for inserting entries into and retrieving information from an embedding object memory, the method comprising:
receiving a content item, the content item having one or more content data;
providing one of the content data associated with the content item to one or more semantic embedding models, wherein the one or more semantic embedding models generate one or more semantic embeddings;
receiving, from one or more of the semantic embedding models, one or more semantic embeddings, wherein a collection of semantic embeddings is associated with a first semantic embedding model of the one or more semantic embedding models, wherein the collection of semantic embeddings comprises a first semantic embedding generated by the first semantic embedding model for at least one content data from the respective content item, wherein the one or more semantic embedding models comprise a version, and wherein each of the semantic embeddings generated by each of the respective one or more semantic embedding models comprise metadata corresponding to the version;
inserting the one or more semantic embeddings into the embedding object memory, wherein the embedding object memory stores one or more semantic embeddings from the collection of semantic embeddings, and wherein the one or more semantic embeddings are associated with a respective indication corresponding to a reference to source data associated with the one or more semantic embeddings;
providing an updated semantic embedding model to replace at least one of the semantic embedding models, the updated semantic embedding model comprising an updated version that is different than the version of the at least one of the semantic embedding models;
receiving, from the updated semantic embedding model, an updated one or more semantic embeddings corresponding to the one or more semantic embeddings generated by the at least one of the semantic embedding models, wherein the updated one or more semantic embeddings are generated based on the one of the content data used to generate the one or more semantic embeddings;
inserting the updated semantic embeddings in the embedding object memory with metadata corresponding to the updated version;
receiving an input embedding;
retrieving a plurality of collections of stored semantic embeddings, from the semantic object memory, based on the input embedding;
retrieving a subset of semantic embeddings from at least one of the plurality of collections of stored semantic embeddings based on a similarity to the input embedding; and
providing the subset of semantic embeddings as an output.
16. The method of claim 15 , wherein the plurality of collections of stored semantic embeddings are retrieved based on the metadata corresponding to the version.
17. The method of claim 15 , further comprising:
determining, based on the subset of semantic embeddings and the input embedding, an action; and
adapting a computing device to perform the action.
18. The method of claim 15 , wherein the retrieving a subset of semantic embeddings comprises:
determining a respective similarity between the input embedding and each semantic embedding of the plurality of collections of stored semantic embeddings;
determining that one or more of the similarities are less than a predetermined threshold; and
retrieving the subset of semantic embeddings with similarities to the input embedding that are less than the predetermined threshold, thereby retrieving subsets of semantic embeddings from at least one of the collections of semantic embeddings that are determined to be related to the input embedding.
19. The method of claim 15 , wherein the semantic embedding models comprise a generative large language model (LLM).
20. The method of claim 15 , wherein the content data are provided to the one or more semantic embedding models locally or via an application programming interface (API).