Granted Patent B2

US 12,216,705 · App. 17/404,367 · Granted Feb 4, 2025

Inventors: Li Huang (Redmond, WA); Meenaz Merchant (Redmond, WA); Houdong Hu (Redmond, WA); Arun Sacheti (Redmond, WA)

Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC

G06F16/583G06F16/24578G06F16/248G06F16/51G06F16/532G06F16/56G06F18/22G06N3/04G06N3/08

Abstract

Claims (53)

1. A computing system, comprising:

a hardware processor; and

at least one memory device storing instructions that, when executed by the hardware processor, cause the hardware processor to perform acts comprising:

obtaining an image, wherein the image is displayed to a user;

in response to receiving an indication that an object in the image has been selected by the user, generating a feature vector that represents the object;

determining a category of the object based upon the feature vector;

retrieving a plurality of attributes based upon the category of the object, wherein the plurality of attributes comprise one or more first attributes and one or more second attributes, wherein the one or more first attributes are indicative of a feature of the object and the one or more second attributes are indicative of a feature of at least one other object in the category but are not indicative of the object;

generating a plurality of search results based upon the feature vector and the plurality of attributes; and

causing the plurality of search results to be displayed within a graphical user interface (GUI).

2. The computing system of claim 1 , wherein the plurality of search results comprise images that include objects belonging to the category of the object displayed in the image.

3. The computing system of claim 2 , wherein the plurality of search results further comprise descriptions of each of the objects.

4. The computing system of claim 1 , wherein the plurality of search results include a first image that includes a first object, wherein the first object has the at least one of the one or more second attributes in place of the one or more first attributes.

5. The computing system of claim 1 , wherein the image comprises a plurality of objects, the acts further comprising:

prior to generating the feature vector, identifying the object from amongst the plurality of objects in the image.

6. The computing system of claim 1 , wherein a second object is in the image, wherein generating the feature vector that represents the object in the image comprises:

cropping the image to generate a query image, wherein the query image includes the object, wherein the query image excludes the second object, wherein the feature vector is generated based upon the query image.

7. The computing system of claim 1 , the acts further comprising:

obtaining user preferences of the user; and

subsequent to generating the plurality of search results and prior to causing the plurality of search results to be displayed within the GUI, assigning rankings to the plurality of search results based upon the user preferences, wherein the plurality of search results are displayed in accordance with the rankings.

8. The computing system of claim 1 , wherein generating the plurality of search results based upon the feature vector and the plurality of attributes comprises:

identifying a database from amongst a plurality of databases based upon the category of the object; and

executing a search over the database based upon the feature vector and the plurality of attributes, wherein the plurality of search results are generated from the search.

9. The computing system of claim 1 , wherein the image is obtained from a computing device operated by the user, wherein the GUI is presented on a display of the computing device.

10. The computing system of claim 1 , wherein the image is obtained from a server computing device.

11. The computing system of claim 1 , wherein retrieving the plurality of attributes comprises executing a rule assigned to the category.

12. A method executed by a processor of a computing system, the method comprising:

obtaining an image, wherein the image is displayed on a display of a computing device operated by a user;

identifying a plurality of objects in the image;

upon identifying the plurality of objects in the image, determining respective categories of each of the plurality of objects in the image;

in response to receiving an indication that an object in the plurality of objects in the image has been selected by the user, generating a feature vector that represents the object;

retrieving a plurality of attributes based upon a category of the object, wherein the plurality of attributes comprise one or more first attributes and one or more second attributes, wherein the one or more first attributes are indicative of a feature of the object and the one or more second attributes are indicative of a feature of at least one other object in the category but are not indicative of the object;

generating a plurality of search results based upon the feature vector and the plurality of attributes; and

causing the plurality of search results to be displayed within a graphical user interface (GUI) shown on the display.

13. The method of claim 12 , wherein each of the plurality of objects in the image is marked with a respective icon, wherein the object is selected when an icon assigned to the object is selected by the user.

14. The method of claim 12 , wherein generating the plurality of search results based upon the feature vector and the plurality of attributes comprises:

generating a visual-word-quantized representation of the object based upon the feature vector;

generating a product quantized representation of object based upon the visual-word-quantized representation of the object; and

executing a search over at least one database based upon the product quantized representation of the object and the attribute set, wherein the plurality of search results are generated based upon the search.

15. The method of claim 12 , further comprising:

obtaining user preferences of the user; and

subsequent to generating the plurality of search results and prior to causing the plurality of search results to be displayed within the GUI, assigning rankings to the plurality of search results based upon the user preferences, wherein the plurality of search results are displayed in accordance with the rankings.

16. The method of claim 12 , wherein generating the feature vector that represents the object in the image comprises:

cropping the image to generate a query image, wherein the query image includes the object, wherein the feature vector is generated based upon the query image.

17. The method of claim 12 , wherein the plurality of search results comprise images that include objects that belong to the category of the object displayed in the image.

18. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processor of a computing system, cause the processor to perform acts comprising:

obtaining an image, wherein the image is displayed within a graphical user interface (GUI) shown on a display of a computing device;

in response to receiving an indication that an object in the image has been selected by a user of the computing device, generating a feature vector that represents the object;

determining a category of the object based upon the feature vector;

retrieving a plurality of attributes based upon the category of the object, wherein the plurality of attributes comprise one or more first attributes and one or more second attributes, wherein the one or more first attributes are indicative of a feature of the object and the one or more second attributes are indicative of at least one other object in the category but are not indicative of the object;

generating a plurality of search results based upon the feature vector and the plurality of attributes; and

transmitting the plurality of search results to the computing device, wherein the plurality of search results are displayed within the GUI.

19. The non-transitory computer-readable storage medium of claim 18 , wherein the image is displayed at a first region of the GUI, wherein the plurality of search results are displayed at a second region of the GUI.

20. The non-transitory computer-readable storage medium of claim 18 , wherein the at least one additional attribute belongs to the category.

Assignments (1)

Continuity (2)

References Cited (61)