Systems and methods for contextualized visual search

1. A system comprising:

one or more computer devices that implement a contextualized visual search (CVS) system, configured to:

receive, via a user interface, a query image and a search request to search for items that display a pictogram that includes the query image;

convert, using a first machine learning model, the query image into a query feature vector;

analyze, using a second machine learning model, target feature vectors of target images of a plurality of items, wherein the second machine learning model is trained to examine different sub-regions of a given target image to (a) determine whether individual ones of the sub-regions contain a view of the query image and (b) identify one or more locations of one or more views of the query image in the given target image;

identify, based at least in part on the analysis, a set of matching images from the target images that contain views of the query image and a set of matching items associated with the matching images; and

output search results via the user interface, wherein the search results indicate the matching items, the matching images, and locations of the views of the query image in the matching images.

2. The system of claim 1 , wherein:

the items are products listed on an e-commerce web portal; and

the user interface is a web interface that displays information about the products.

3. The system of claim 1 , wherein the CVS system is configured to:

prior to receiving the search request:

generate the target feature vectors from the target images using the first machine learning model; and

store the target feature vectors in a data store in shards organized by item categories.

4. The system of claim 3 , wherein the CVS system is configured to:

receive a request to add a new item to the CVS system, wherein the request indicates a new image of the new item and an item category of the new item;

generate a new target feature vector for the new image; and

store a record of the new image in the data store, wherein the record includes an identifier of the new item, the item category of the new item, and the new target feature vector.

5. The system of claim 1 , wherein:

the search request is specified by a user; and

the CVS system configured to:

determine, prior to the search request, one or more item categories relevant to the user based on user features of the user and using an item category prediction model;

populate an in-memory cache with target feature vectors associated with the one or more item categories; and

perform the search request using the target feature vectors in the in-memory cache.

6. A method comprising:

performing, by a contextualized visual search (CVS) system implemented by one or more computers:

receiving a query image and a search request to search for items that display a pictogram that includes the query image;

converting, using a first machine learning model, the query image into a query feature vector;

analyzing, using a second machine learning model, target feature vectors of target images of a plurality of items, wherein the second machine learning model is trained to examine different sub-regions of a given target image to (a) determine whether individual ones of the sub-regions contain a view of the query image and (b) identify one or more locations of one or more views of the query image in the given target image;

identifying, based at least in part on the analysis, a set of matching images from the target images that contain views of the query image and a set of matching items associated with the matching images; and

outputting search results that indicate the matching items, the matching images, and locations of the views of the query image in the matching images.

7. The method of claim 6 , further comprising performing, by the CVS system:

detecting a user accessing the CVS system prior to the user issuing the search request;

determining one or more item categories relevant to the user based on user features of the user and using an item category prediction model;

populating an in-memory cache with target feature vectors associated with the one or more item categories; and

performing the search request using the target feature vectors in the in-memory cache.

8. The method of claim 6 , wherein:

the search request is issued by a user via an interactive user interface;

the search results are outputted via the interactive user interface; and

the matching items in the search results are ranked based at least in part on user features of the user.

9. The method of claim 8 , further comprising performing, by the CVS system:

receiving one or more actions of the user on the search results; and

updating the user features of the user based at least in part on the one or more actions.

10. The method of claim 6 , wherein:

the search request is received by a request gateway service configured to call a plurality of backend services including a feature converter service and a feature matching service;

the feature converter service is configured to perform the conversion of the query image; and

the feature matching service is configured to perform the analyzing of the target feature vectors.

11. The method of claim 6 , further comprising performing, by the CVS system:

prior to receiving the search request:

generating the target feature vectors from the target images using the first machine learning model; and

storing the target feature vectors in a data store in shards organized by item categories.

12. The method of claim 11 , further comprising performing, by the CVS system:

receiving a request to add a new item to the CVS system, wherein the request indicates an image of the new item and an item category of the new item;

generating a new target feature vector for the image; and

storing a record of the image in the data store, wherein the record includes an identifier of the new item, the item category of the new item, and the new target feature vector.

13. The method of claim 6 , wherein:

the first and second machine learning models are neural networks; and

the method comprises:

training the first and second machine learning models together as a one-shot object detection network; and

deploying the first and second machine learning models to implement different components of the CVS system after the training.

14. The method of claim 13 , wherein:

the one-shot object detection network receives an input query image and an input target image and includes a learnable Robust Feature Matching module that outputs an emphasized feature vector that emphasizes sub-regions in the input target image that have similar features to the input query image.

15. The method of claim 13 , wherein:

the one-shot object detection network includes one or more transformers that performs a set of positional object queries on different sub-regions of the input target image to detect the input query image in the different sub-regions.

16. The method of claim 13 , further comprising:

obtaining a dataset of target images with annotated regions of interest;

cropping out the regions of interest as query images;

randomly generating query-target pairs from the query images and target images; and

using the query-target pairs as a training dataset to train the one-shot object detection network.

17. The method of claim 16 , further comprising, prior to the generating of the query-target pairs:

generating and adding synthetic images to the dataset of target images, wherein the generating includes (a) stitching together multiple ones of the target images or (b) pasting regions of interest onto one or more random backgrounds.

18. One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors implement a contextualized visual search (CVS) system and cause the CVS system to:

receive a query image and a search request to search for items that display a pictogram that includes the query image;

convert, using a first machine learning model, the query image into a query feature vector;

analyze, using a second machine learning model, target feature vectors of target images of a plurality of items, wherein the second machine learning model is trained to examine different sub-regions of a given target image to (a) determine whether individual ones of the sub-regions contain a view of the query image and (b) identify one or more locations of one or more views of the query image in the given target image;

identify, based at least in part on the analysis, a set of matching images from the target images that contain views of the query image and a set of matching items associated with the matching images; and

output search results that indicate the matching items, the matching images, and locations of the views of the query image in the matching images.

19. The one or more non-transitory computer-accessible storage media of claim 18 , wherein the pictogram is one or more of an icon, a logo, a sign, a mark, a label, and a symbol.

20. The one or more non-transitory computer-accessible storage media of claim 19 , wherein the program instructions when executed on or across the one or more processors cause the CVS system to:

rank matching items in the search results based at least in part on the user features.