Searching augmented reality experiences using visual embeddings
Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for performing operations comprising: receiving an image from a client device; applying a machine learning model to the image to generate an embedding query vector, the machine learning model being trained to encode a plurality of images and text into a common embedding space; searching, based on the embedding query vector, a database of augmented reality (AR) experiences to identify a subset of AR experiences associated with one or more embeddings that correspond to the embedding query vector; and transmitting to the client device the subset of AR experiences associated with the one or more embeddings that correspond to the embedding query vector.
1. A method comprising:
receiving, by one or more processors, an image from a device;
applying a machine learning model to the image to generate an embedding query vector, the machine learning model trained to encode a plurality of images and text into a common embedding space;
searching, based on the embedding query vector, a database of augmented reality (AR) experiences to identify a subset of AR experiences associated with one or more embeddings that correspond to the embedding query vector, the subset of AR experiences being identified by:
computing a Gaussian distance metric between the embedding query vector and each of a plurality of indices; and
determining that nearest neighbor indices are associated with a Gaussian distance metric that corresponds to a similarity threshold; and
transmitting to the device the subset of AR experiences associated with the one or more embeddings that correspond to the embedding query vector.
2. The method of claim 1 , wherein the device generates a user interface that presents the subset of AR experiences.
3. The method of claim 2 , further comprising causing the device to modify the image based on an AR experience selected from the user interface that presents the subset of AR experiences.
4. The method of claim 1 , further comprising:
identifying the nearest neighbor indices for the embedding query vector; and
selecting the subset of AR experiences in response to determining that the subset of AR experiences are associated with the nearest neighbor indices.
5. The method of claim 4 , further comprising:
computing a cosine similarity metric between the embedding query vector and each of a plurality of indices; and
determining that the nearest neighbor indices are associated with the cosine similarity metric that corresponds to a similarity threshold.
6. The method of claim 1 , further comprising:
presenting a user interface to a first user of a plurality of users;
receiving a textual search query from the first user via the user interface, the textual search query comprising a request to identify one or more AR experiences;
presenting, in the user interface, search results in response to receiving the textual search query, the search results comprising a plurality of AR experiences;
monitoring interactions of the first user with the user interface to collect metadata that identifies the interactions indicating whether the first user has interacted with a given AR experience of the plurality of AR experiences;
storing each of the interactions in the metadata together with the textual search query as part of a search log in association with the given AR experience.
7. The method of claim 6 , further comprising:
accessing the search log that includes a plurality of search queries for AR experiences received previously from a plurality of users;
identifying the given AR experience that was selected in response to a first search query of the plurality of search queries; and
storing a first set of text of the first search query in association with the given AR experience.
8. The method of claim 7 , further comprising:
determining that the given AR experience was selected in response to a second search query of the plurality of search queries; and
storing a second set of text of the second search query in association with the given AR experience.
9. The method of claim 7 , further comprising repeating the identifying and storing for each of the plurality of search queries to generate the database that associates each of a plurality of AR experiences with a respective set of text of the plurality of search queries.
10. The method of claim 9 , further comprising applying the machine learning model to a set of text associated with each of the plurality of AR experiences of the database to generate an AR experience embedding for each of the plurality of AR experiences.
11. The method of claim 10 , wherein searching the database of AR experiences comprising searching the AR experience embedding for each of the plurality of AR experiences to identify the subset of AR experiences that are associated with AR experience embeddings that correspond to the embedding query vector.
12. The method of claim 9 , further comprising removing non-ASCII letter search queries from the plurality of search queries.
13. The method of claim 9 , further comprising removing search queries from the plurality of search queries that fail to correspond to a length criteria.
14. The method of claim 1 , further comprising ranking the subset of AR experiences based on one or more criteria.
15. The method of claim 1 , further comprising:
obtaining one or more tags associated with the subset of AR experiences;
determining that a set of tags of the one or more tags associated with an individual AR experience of the subset of AR experiences corresponds to a blocked category; and
removing the individual AR experience from the subset of AR experiences in response to determining that the set of tags associated with the individual AR experience corresponds to the blocked category.
16. The method of claim 1 , further comprising training the machine learning model by performing operations comprising:
receiving training data comprising a plurality of image-text pairs;
selecting a first image-text pair from the plurality of image-text pairs;
applying the machine learning model to the first image-text pair to relate visual embeddings of an image of the first image-text pair and textual embeddings of text of the first image-text pair with a same embedding in the common embedding space; and
repeating the selecting and applying operations for each of the plurality of image-text pairs.
17. The method of claim 1 , wherein the machine learning model comprises a network that indicates correspondence between a given image and given text.
18. A system comprising:
at least one processor configured to perform operations comprising:
receiving an image from a device;
applying a machine learning model to the image to generate an embedding query vector, the machine learning model being trained to encode a plurality of images and text into a common embedding space;
searching, based on the embedding query vector, a database of augmented reality (AR) experiences to identify a subset of AR experiences associated with one or more embeddings that correspond to the embedding query vector, the subset of AR experiences being identified by:
computing a Gaussian distance metric between the embedding query vector and each of a plurality of indices; and
determining that nearest neighbor indices are associated with a Gaussian distance metric that corresponds to a similarity threshold; and
transmitting to the device the subset of AR experiences associated with the one or more embeddings that correspond to the embedding query vector.
19. The system of claim 18 , wherein the device generates a user interface that presents the subset of AR experiences.
20. A non-transitory machine-readable storage medium that includes instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
receiving an image from a device;
applying a machine learning model to the image to generate an embedding query vector, the machine learning model being trained to encode a plurality of images and text into a common embedding space;
searching, based on the embedding query vector, a database of augmented reality (AR) experiences to identify a subset of AR experiences associated with one or more embeddings that correspond to the embedding query vector, the subset of AR experiences being identified by:
computing a Gaussian distance metric between the embedding query vector and each of a plurality of indices; and
determining that nearest neighbor indices are associated with a Gaussian distance metric that corresponds to a similarity threshold; and
transmitting to the device the subset of AR experiences associated with the one or more embeddings that correspond to the embedding query vector.