IP Library › Granted Patent US 11,681,919
Granted Patent B2
US 11,681,919 · App. 17/331,161 · Granted Jun 20, 2023

Automatically selecting query objects in digital images

Inventors: Khoi Pham (Hyattsville, MD); Scott Cohen (Sunnyvale, CA); Zhe Lin (Fremont, CA); Zhihong Ding (Fremont, CA); Walter Wei Tuh Chang (San Jose, CA)
Assignee: Adobe Inc.
G06N3/08G06F18/2113G06F18/2155G06F18/2163G06V10/764G06V10/765G06V10/771G06V10/7753G06V10/82
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,681,919
App. No.
17/331,161
Granted
Jun 20, 2023
Kind
B2
Abstract

The present disclosure relates to an object selection system that automatically detects and selects objects in a digital image utilizing a large-scale object detector. For instance, in response to receiving a request to automatically select a query object with an unknown object class in a digital image, the object selection system can utilize a large-scale object detector to detect potential objects in the image, filter out one or more potential objects, and label the remaining potential objects in the image to detect the query object. In some implementations, the large-scale object detector utilizes a region proposal model, a concept mask model, and an auto tagging model to automatically detect objects in the digital image.

Claims (58)

1. In a digital medium environment for creating or editing digital images, a computer-implemented method of selecting query objects, comprising:

in response to receiving a query string to select a query object within a digital image, generating a plurality of region proposals for objects detected in the digital image utilizing a region proposal model;

generating a heat mask of the digital image that indicates areas in the digital image that potentially contain the query object;

adding an additional region proposal to the plurality of region proposals based on identifying an area in the heat mask that is missing a region proposal from the plurality of region proposals;

generating object labels for the plurality of region proposals; and

detecting the query object based on comparing the object labels generated for the plurality of region proposals with the query string.

2. The computer-implemented method of claim 1 , further comprising adding the additional region proposal to the plurality of region proposals by aligning the plurality of region proposals with the heat mask to identify an additional area in the heat mask for which no region proposal exists.

3. The computer-implemented method of claim 1 , further comprising identifying the area in the heat mask for which no region proposal exists by determining a threshold number of positive pixels in the heat mask that do not align with one or more region proposals of the plurality of region proposals.

4. The computer-implemented method of claim 1 , further comprising:

dividing the digital image into a plurality of sub-images; and

utilizing the region proposal model to identify the plurality of region proposals by detecting region proposals within each of the plurality of sub-images.

5. The computer-implemented method of claim 1 , further comprising generating object labels for region proposals utilizing an auto-tagging model.

6. The computer-implemented method of claim 5 , wherein:

the query object comprises a noun included within the query string; and

detecting the query object comprises comparing the object labels generated for the plurality of region proposals with the query string to determine a match between the noun and an object label of the object labels.

7. The computer-implemented method of claim 1 , further comprising generating a subset of filtered region proposals by filtering the plurality of region proposals based on the heat mask to remove a subset of region proposals from the region proposals.

8. The computer-implemented method of claim 7 , wherein filtering the plurality of region proposals based on the heat mask to identify the subset of filtered region proposals comprises aligning the plurality of region proposals with the heat mask to identify region proposals that overlap with areas identified in the heat mask as potentially containing the query object to a satisfaction of an overlapping object threshold.

9. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computing device to:

in response to receiving a query string indicating a query object to select within a digital image, divide the digital image into a plurality of sub-images;

for each sub-image of the plurality of sub-images:

generate a plurality of region proposals for potential objects in the sub-image; and

generate an object label for each region proposal in the sub-image; and

detect, within one or more of the plurality of sub-images, the query object from the plurality of region proposals based on the object labels.

10. The non-transitory computer-readable medium of claim 9 , wherein the instructions that, when executed by the at least one processor, cause the computing device to determine a number of the plurality of sub-images based on dimensions of the digital image.

11. The non-transitory computer-readable medium of claim 9 , further comprising additional instructions that, when executed by the at least one processor, cause the computing device to:

generate a concept mask, utilizing a concept mask model, that indicates areas that likely include the query object indicated in the query string; and

add an additional region proposal to the plurality of region proposals in the sub-image based on the concept mask.

12. The non-transitory computer-readable medium of claim 9 , wherein the instructions that, when executed by the at least one processor, cause the computing device to:

filter the plurality of region proposals within each sub-image to identify a subset of filtered region proposals; and

generate the object label for each filtered region proposal of the subset of filtered region proposals in the sub-image utilizing an auto-tagging model.

13. The non-transitory computer-readable medium of claim 12 , wherein the instructions that, when executed by the at least one processor, cause the computing device to generate the plurality of region proposals for potential objects in the sub-image by utilizing a region proposal model that generates approximate boundaries for each detected object within the sub-image.

14. The non-transitory computer-readable medium of claim 13 , wherein the instructions that, when executed by the at least one processor, cause the computing device to identify the subset of filtered region proposals by generating a heat mask comprising areas in the sub-image that potentially comprise the query object.

15. The non-transitory computer-readable medium of claim 12 , wherein the instructions that, when executed by the at least one processor, cause the computing device to identify the subset of filtered region proposals by utilizing a concept mask model to generate a concept mask comprising areas in the sub-image that potentially comprise the query object.

16. A system comprising:

one or more memory devices comprising:

a digital image comprising a plurality of objects;

a query string indicating a query object to be automatically selected;

a large-scale object detector comprising a region proposal model and a concept mask model; and

a heat mask of the digital image that indicates areas in the digital image where the query object potentially resides generated by the concept mask model; and

one or more server devices configured to cause the system to:

in response to receiving the query string, utilizing the region proposal model to identify a plurality of region proposals for objects detected within the digital image;

determine a missing potential object within the digital image based on comparing the plurality of region proposals to the heat mask;

add an additional region proposal for the missing potential object to the plurality of region proposals; and

detect the query object from the plurality of region proposals based on the plurality of region proposals.

17. The system of claim 16 , wherein the one or more server devices are configured to cause the system to:

divide the digital image into a plurality of sub-images; and

utilize the region proposal model to identify the plurality of region proposals by detecting region proposals within each of the plurality of sub-images.

18. The system of claim 16 , wherein the one or more server devices are configured to cause the system to:

generate object labels for the plurality of region proposals utilizing an auto-tagging model; and

detect the query object from the plurality of region proposals based on comparing the query string with the object labels generated for the plurality of region proposals.

19. The system of claim 18 , wherein the one or more server devices are configured to cause the system to identify the plurality of region proposals for objects detected within the digital image utilizing the region proposal model by:

detecting a plurality of potential objects within the digital image;

generating a potential object score for each of the plurality of potential objects; and

selecting each potential object from the plurality of potential objects that has a potential object score that satisfies a minimum potential object threshold score to include within the plurality of region proposals.

20. The system of claim 19 , further comprising the one or more server devices configured to cause the system to:

filter one or more of the plurality of region proposals based on the heat mask to identify a subset of filtered region proposals, and wherein:

generate the object labels for each filtered region proposal of the subset of filtered region proposals; and

detect the query object based on comparing the object labels generated for each filtered region proposal of the subset of filtered region proposals with the query string.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 26, 2021
From: PHAM, KHOI; COHEN, SCOTT; LIN, ZHE; DING, ZHIHONG; CHANG, WALTER WEI TUH
To: ADOBE INC.
Reel/Frame 056361/0566 →
Continuity (2)
Continuation 16817418 · Mar 12, 2020
Related Publication 20210319255A1 · Oct 14, 2021
Cited By (2)
US 12,283,052 US 12,525,027