Three-dimensional object identification using two-dimensional image data
Various approaches provide for visual similarity-based search techniques that allow a user to provide two-dimensional image data (e.g., still images or video) about an item and search for items having related three-dimensional features. In order to create an electronic catalog of items that is searchable by three-dimensional features, the features are extracted from image data for each item. The three-dimensional features can be extracted using a trained model, such as a trained neural network or other machine learning-based approach. Thus, three-dimensional features of items in the electronic catalog can be determined and used to select or rank the items in response to a visual search query based on determined three-dimensional features of the query item. For example, three-dimensional features of the query item can be used to query the electronic catalog of items, in which items having visual attributes of similar three-dimensional features are selected and returned as search results.
1. A computer-implemented method, comprising:
obtaining a two-dimensional query image including a representation of a three-dimensional item;
processing the two-dimensional query image using a trained model to determine three-dimensional features of the item, the trained model including at least one neural network configured to apply at least one three-dimensional feature extractor and generate three-dimensional feature data, the trained model further combining the three-dimensional feature data into a single representation and generating one or more three-dimensional models of the item;
using the trained model to determine similarity scores between the one or more three-dimensional models of the item and three-dimensional feature data for a plurality of items, wherein the similarity scores are determined based on at least one of rank position, distance, or another measure derived from the rank position or distance;
determining whether the similarity scores satisfy a threshold; and
providing content, for display by an interface on a client device, for at least one item of the plurality based at least in part upon the similarity scores.
2. The computer-implemented method of claim 1 , further comprising:
obtaining three-dimensional representations of items in a category of items having item information stored in an electronic catalog, the three-dimensional representations generated using a plurality of images representative of the items from different viewpoints; and
using the three-dimensional representations to train a convolutional neural network to generate the trained model, for the category of items, to generate three-dimensional feature information from image data that includes a two-dimensional representation of an item.
3. The computer-implemented method of claim 1 , further comprising:
processing a catalog of images of items offered through an electronic marketplace using the trained model to determine the three-dimensional feature data; and
storing the three-dimensional feature data in a database associated with the electronic marketplace.
4. The computer-implemented method of claim 1 , wherein the trained model is generated using a processing module and, and wherein the two-dimensional query image includes a first image having a first viewpoint of the item and a second image having a second viewpoint of the item, the method further comprising:
processing the first image and the second image using the processing module to determine respective first and second three-dimensional features.
5. The computer-implemented method of claim 4 , further comprising:
processing the first three-dimensional features and the second three-dimensional features using a second trained model to combine the first three-dimensional features of the first viewpoint and the second three-dimensional features of the second viewpoint to generate combined three-dimensional features.
6. The computer-implemented method of claim 5 , further comprising:
processing the combined three-dimensional features using at least one transform to reduce a number of dimensions of the combined three-dimensional features.
7. The computer-implemented method of claim 5 , further comprising:
processing the combined three-dimensional features using a third trained model to determine a three-dimensional representation of the item.
8. The computer-implemented method of claim 7 , further comprising:
comparing the three-dimensional representation of the item to a catalog three-dimensional feature data for the item to determine a similarity score; and
assigning a level of confidence to the trained model based at least in part upon on the similarity score.
9. The computer-implemented method of claim 1 , wherein the two-dimensional query image is a first query image, the method further comprising:
determining a second query image including a second representation of the item from a second viewpoint; and
processing the first query image and the second query image using the trained model to determine combined three-dimensional features, wherein the similarity scores are based at least in part upon the combined three-dimensional features.
10. The computer-implemented method of claim 1 , wherein the plurality of items include products offered through an electronic marketplace, and wherein the three-dimensional features of the query image correspond to a pose and a three-dimensional shape of the item.
11. The computer-implemented method of claim 1 , wherein the two-dimensional query image is uploaded from the client device, captured using a camera of the client device, or pointed to via a link provided from the client device, and wherein the two-dimensional query image includes at least one viewpoint of the item.
12. The computer-implemented method of claim 1 , further comprising:
wherein the similarity scores are further determined based on K-nearest neighbor distances or rankings positions.
13. A system, comprising:
at least one computing device processor; and
a memory device including instructions that, when executed by the at least one computing device processor, cause the system to:
obtain a two-dimensional query image including a representation of a three-dimensional item;
process the two-dimensional query image using a trained model to determine three-dimensional features of the item, the trained model including at least one neural network configured to apply at least one three-dimensional feature extractor and generate three-dimensional feature data, the trained model further combining the three-dimensional feature data into a single representation and generating one or more three-dimensional models of the item;
use the trained model to determine similarity scores between the one or more three-dimensional models of the item and three-dimensional feature data for a plurality of items, wherein the similarity scores are determined based on at least one of rank position, distance, or another measure derived from the rank position or distance;
determine whether the similarity scores satisfy a threshold; and
provide content, for display by an interface on a client device, for at least one item of the plurality based at least in part upon the similarity scores.
14. The system of claim 13 , wherein the instructions when executed further cause the system to:
obtain three-dimensional representations of items in a category of items having item information stored in an electronic catalog, the three-dimensional representations generated using a plurality of images representative of the items from different viewpoints; and
use the three-dimensional representations to train a convolutional neural network to generate the trained model, for the category of items, to generate three-dimensional feature information from image data that includes a two-dimensional representation of an item.
15. The system of claim 14 , wherein the instructions when executed further cause the system to:
process a catalog of images of items offered through an electronic marketplace using the trained model to determine the three-dimensional feature data; and
store the three-dimensional feature data in a database associated with the electronic marketplace.
16. The system of claim 13 , wherein the trained model is generated using a processing module, and wherein the two-dimensional query image includes a first image having a first viewpoint of the item and a second image having a second viewpoint of the item, and wherein the instructions when executed further cause the system to:
process the first image and the second image using the processing module to determine respective first and second three-dimensional features.
17. The system of claim 16 , wherein the instructions when executed further cause the system to:
process the first three-dimensional features and the second three-dimensional features using a second trained model to combine the first three-dimensional features of the first viewpoint and the second three-dimensional features of the second viewpoint to generate combined three-dimensional features.
18. The system of claim 13 , wherein the instructions when executed further cause the system to:
further determine the similarity scores based on K-nearest neighbor distances or rankings positions.