IP Library › Granted Patent US 12,748,803
Granted Patent B2
US 12,748,803 · App. 17/479,172 · Granted Sep 29, 2026

Cognitive image searching based on personalized image components of a composite image

Inventors: Swaminathan Balasubramanian (Troy, MI); Radha Mohan De (Howrah, IN); Mamnoon Jamil (Voorhees, NJ); Jian Wu (Mansfield, TX); Cheranellore Vasudevan (Bastrop, TX)
Assignee: International Business Machines Corporation
G06F16/53G06F16/58
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,748,803
App. No.
17/479,172
Granted
Sep 29, 2026
Kind
B2
Abstract

Embodiments of the invention are directed to a computer-implemented method of performing an electronic search. The computer-implemented method includes receiving, using a processor, a composite electronic image including a plurality of electronically identifiable objects, wherein the composite electronic image is associated with a user. The processor is used to segment the composite electronic image into sub-images by providing at least one of the sub-images for each of the plurality of electronically identifiable objects. For each of the sub-images, the processor is used to perform personalized sub-image search operations. The personalized sub-image search operations include selecting a sub-image-to-be-searched from among the sub-images; associating the sub-image-to-be-searched with personalized metadata of the user; and searching, based at least in part on the personalized metadata of the user, a database to return a set of search images.

Claims (63)

1 . A computer-implemented method of performing an electronic search, the computer-implemented method comprising:

receiving, using a processor, a composite electronic image comprising a plurality of electronically identifiable objects, wherein the composite electronic image is associated with a user;

segmenting, using the processor, the composite electronic image into sub-images by providing at least one of the sub-images for each of the plurality of electronically identifiable objects; and

using the processor to perform personalized sub-image search operations on each of the sub-images without relying on the user taking action, the personalized sub-image search operations comprising:

deciding, by the processor, a sub-image-to-be-searched from among the sub-images;

associating the sub-image-to-be-searched with personalized metadata of the user; and

searching, based at least in part on the personalized metadata of the user, a database to return a set of search images.

2 . The computer-implemented method of claim 1 , wherein the personalized metadata of the user comprises:

a description of a relevance of the sub-image-to-be-searched to the user;

a relative size of the sub-image-to-be-searched to the composite image; and

a relative position of the sub-image-to-be-searched within the composite image.

3 . The computer-implemented method of claim 2 , wherein the personalized metadata of the user is derived from a corpus of information about the user.

4 . The computer-implemented method of claim 1 , wherein the personalized sub-image search operations further comprise:

assigning a relevance level to the sub-image-to-be-searched;

wherein the relevance level comprises a relevance of the sub-image-to-be-searched to the user; and

ranking the sub-image-to-be-searched against the sub-images based at least in part on the relevance level.

5 . The computer-implemented method of claim 1 , wherein the personalized sub-image search operations further comprise:

presenting results of the personalized sub-image search operations to the user; and

receiving feedback from the user on the results of the personalized sub-image search operations.

6 . The computer-implemented method of claim 5 further comprising repeating the personalized sub-image search operations for the sub-image-to-be-searched based on the feedback.

7 . The computer-implemented method of claim 4 , wherein the processor comprises a classifier trained to:

generate the personalized metadata of the user based on the corpus of information about the user; and

determine the relevance of the sub-image-to-be-searched to the user based at least in part on the corpus of information about the user.

8 . A computer system for performing an electronic search, the computer system comprising a memory communicatively coupled to a processor, the processor configured to perform processor operations comprising:

receiving a composite electronic image comprising a plurality of electronically identifiable objects, wherein the composite electronic image is associated with a user;

segmenting the composite electronic image into sub-images by providing at least one of the sub-images for each of the plurality of electronically identifiable objects; and

performing personalized sub-image search operations on each of the sub-images without relying on the user taking action, the personalized sub-image search operations comprising:

deciding, by the processor, a sub-image-to-be-searched from among the sub-images;

using the processor to perform a cognitive task comprising determining a level of relevance of the sub-image-to-be-searched to the user, incorporating the level of relevance of the sub-image-to-be-searched to the user into metadata of the sub-image-to-be-searched to create personalized metadata of the user, and associating the sub-image-to-be-searched with the personalized metadata of the user; and

searching, based at least in part on the personalized metadata of the user, a database to return a set of search images.

9 . The computer system of claim 8 , wherein the personalized metadata of the user comprises a description of a relevance of the sub-image-to-be-searched to the user.

10 . The computer system of claim 9 , wherein the personalized metadata of the user is derived from a corpus of information about the user.

11 . The computer system of claim 8 , wherein the personalized sub-image search operations further comprise:

assigning a relevance level to the sub-image-to-be-searched;

wherein the relevance level comprises the relevance of the sub-image-to-be-searched to the user; and

ranking the sub-image-to-be-searched against the sub-images based at least in part on the relevance level.

12 . The computer system of claim 8 , wherein the personalized sub-image search operations further comprise:

presenting results of the personalized sub-image search operations to the user; and

receiving feedback from the user on the results of the personalized sub-image search operations.

13 . The computer system of claim 12 , wherein the processor operations further comprise repeating the personalized sub-image search operations for the sub-image-to-be-searched based on the feedback.

14 . The computer system of claim 11 , wherein the processor comprises a classifier trained to:

generate the personalized metadata of the user based on the corpus of information about the user; and

determine the relevance of the sub-image-to-be-searched to the user based at least in part on the corpus of information about the user.

15 . A computer program product for performing an electronic search, the computer program product comprising a computer readable program stored on a computer readable storage medium, wherein the computer readable program, when executed on a processor, causes the processor to perform a method comprising:

receiving a composite electronic image comprising a plurality of electronically identifiable objects, wherein the composite electronic image is associated with a user;

segmenting the composite electronic image into sub-images by providing at least one of the sub-images for each of the plurality of electronically identifiable objects; and

performing personalized sub-image search operations on each of the sub-images without relying on the user taking action, the personalized sub-image search operations comprising:

deciding, by the processor, a sub-image-to-be-searched from among the sub-images;

associating the sub-image-to-be-searched with personalized metadata of the user; and

searching, based at least in part on the personalized metadata of the user, a database to return a set of search images.

16 . The computer program product of claim 15 , wherein the personalized metadata of the user comprises a description of a relevance of the sub-image-to-be-searched to the user.

17 . The computer program product of claim 16 , wherein the personalized metadata of the user is derived from a corpus of information about the user.

18 . The computer program product of claim 15 , wherein the personalized sub-image search operations further comprise:

assigning a relevance level to the sub-image-to-be-searched;

wherein the relevance level comprises the relevance of the sub-image-to-be-searched to the user; and

ranking the sub-image-to-be-searched against the sub-images based at least in part on the relevance level.

19 . The computer program product of claim 15 , wherein the personalized sub-image search operations further comprise:

presenting results of the personalized sub-image search operations to the user; and

receiving feedback from the user on the results of the personalized sub-image search operations;

wherein the processor operations further comprise repeating the personalized sub-image search operations for the sub-image-to-be-searched based on the feedback.

20 . The computer program product of claim 18 , wherein the processor comprises a classifier trained to:

generate the personalized metadata of the user based on the corpus of information about the user; and

determine a relevance of the sub-image-to-be-searched to the user based at least in part on the corpus of information about the user.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 20, 2021
From: BALASUBRAMANIAN, SWAMINATHAN; DE, RADHA MOHAN; JAMIL, MAMNOON; WU, JIAN; VASUDEVAN, CHERANELLORE
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 057530/0209 →
Continuity (1)
Related Publication 20230093468A1 · Mar 23, 2023
References Cited (103)
US 7657100B2 · Gokturk · 2010 [cited by examiner]
US 7657126B2 · Gokturk · 2010 [cited by examiner]
US 7660468B2 · Gokturk · 2010 [cited by examiner]
US 7680324B2 · Boncyk et al. · 2010 [cited by applicant]
US 7716157B1 · Bourdev et al. · 2010 [cited by applicant]
US 8274523B2 · Ptucha · 2012 [cited by examiner]
US 8345982B2 · Gokturk et al. · 2013 [cited by applicant]
US 8571358B2 · Zhao et al. · 2013 [cited by applicant]
US 8700595B2 · Baird · 2014 [cited by examiner]
US 9008435B2 · Gokturk et al. · 2015 [cited by applicant]
US 9218364B1 · Garrigues · 2015 [cited by examiner]
US 9230223B2 · Cohen · 2016 [cited by examiner]
US 9582156B2 · Battles · 2017 [cited by examiner]
US 10216763B2 · Butterfield · 2019 [cited by examiner]
US 10339404B2 · Greenberger · 2019 [cited by examiner]
US 10467817B1 · Yip · 2019 [cited by examiner]
US 10515110B2 · Jing · 2019 [cited by examiner]
US 10565256B2 · Badr et al. · 2020 [cited by applicant]
US 10684738B1 · Sicora · 2020 [cited by examiner]
US 10726086B2 · Longbin · 2020 [cited by examiner]
US 10789622B2 · Ayush · 2020 [cited by examiner]
US 10909166B1 · Bogazzi · 2021 [cited by examiner]
US 10956967B2 · Ayush · 2021 [cited by examiner]
US 11030240B1 · Janakiraman · 2021 [cited by examiner]
US 11073975B1 · Mueller · 2021 [cited by examiner]
US 11113883B2 · Zia · 2021 [cited by examiner]
US 11127213B2 · Zia · 2021 [cited by examiner]
US 11163819B2 · Canelis · 2021 [cited by examiner]
US 11176189B1 · Hohwald · 2021 [cited by examiner]
US 11210508B2 · Miller · 2021 [cited by examiner]
US 11216506B1 · Ranzinger · 2022 [cited by examiner]
US 11314827B2 · Zwebner · 2022 [cited by examiner]
US 11328346B2 · Ramos · 2022 [cited by examiner]
US 11520949B2 · Reynolds · 2022 [cited by examiner]
US 11644947B1 · Sicora · 2023 [cited by examiner]
US 11681752B2 · Janakiraman · 2023 [cited by examiner]
US 20040071368A1 · Chadha · 2004 [cited by examiner]
US 20060031486A1 · Miner · 2006 [cited by examiner]
US 20080069480A1 · Aarabi · 2008 [cited by examiner]
US 20080082426A1 · Gokturk · 2008 [cited by examiner]
US 20090125482A1 · Peregrine · 2009 [cited by examiner]
US 20090254539A1 · Wen · 2009 [cited by examiner]
US 20110025709A1 · Ptucha · 2011 [cited by examiner]
US 20110029635A1 · Shkurko · 2011 [cited by examiner]
US 20140156704A1 · Jenkins · 2014 [cited by examiner]
US 20140233845A1 · Wagner · 2014 [cited by examiner]
US 20150120719A1 · Aarabi · 2015 [cited by examiner]
US 20150134688A1 · Jing · 2015 [cited by examiner]
US 20170053186A1 · Allen · 2017 [cited by examiner]
US 20180039700A1 · Greenberger · 2018 [cited by examiner]
US 20180089229A1 · Li · 2018 [cited by examiner]
US 20180137201A1 · Longbin · 2018 [cited by examiner]
US 20180189325A1 · Hohwald · 2018 [cited by examiner]
US 20180367835A1 · Hamidi-Rad · 2018 [cited by examiner]
US 20190121879A1 · Canelis · 2019 [cited by examiner]
US 20190272336A1 · Ciecko · 2019 [cited by examiner]
US 20190318405A1 · Hu · 2019 [cited by examiner]
US 20190354609A1 · Huang et al. · 2019 [cited by applicant]
US 20190392055A1 · Rose et al. · 2019 [cited by applicant]
US 20200327149A1 · Brennenstuhl · 2020 [cited by examiner]
US 20200356592A1 · Yada · 2020 [cited by examiner]
US 20210064612A1 · Huang · 2021 [cited by examiner]
US 20210064679A1 · Zwebner · 2021 [cited by examiner]
US 20210073252A1 · Guo · 2021 [cited by examiner]
US 20210073272A1 · Garrett · 2021 [cited by examiner]
US 20210165818A1 · Matsushita · 2021 [cited by applicant]
US 20210256060A1 · Janakiraman · 2021 [cited by examiner]
US 20210256061A1 · Janakiraman · 2021 [cited by examiner]
US 20210294834A1 · Mai · 2021 [cited by examiner]
US 20210294835A1 · Koroa · 2021 [cited by examiner]
US 20220050867A1 · Waller · 2022 [cited by examiner]
US 20220101578A1 · Bedi · 2022 [cited by examiner]
US 20220269895A1 · Barkan · 2022 [cited by examiner]
US 20230093468A1 · Balasubramanian · 2023 [cited by examiner]
US 20230205817A1 · Janakiraman · 2023 [cited by examiner]
US 20260105098A1 · Wu · 2026 [cited by examiner]
CN 112313643A · 2021 [cited by applicant]
CN 117980894A · 2024 [cited by applicant]
EP 4405828A1 · 2024 [cited by applicant]
JP 2012524343A · 2012 [cited by applicant]
JP 2019023923A · 2019 [cited by applicant]
JP 2021086438A · 2021 [cited by applicant]
JP 2021192241A · 2023 [cited by applicant]
JP 2024535035A · 2024 [cited by applicant]
KR 20210005733A · 2021 [cited by applicant]
TW 201610826A · 2016 [cited by applicant]
TW I831229B · 2024 [cited by applicant]
WO 2023041648A1 · 2023 [cited by applicant]
Boland, M. “Google Lens: Organizing the World's Imagery.” AR Insider. https://arinsider.co/2019/01/29/google-lens-organizing-the-worlds-imagery/ (Jan. 29, 2019), 14 pages. [cited by applicant]
Eksombatchai, “An update on Pixie, Pinterest's recommendation system,” Pinterest Engineering Blog, https://medium.com/pinterest-engineering/an-update-on-pixie-pinterests-recommendation-system-6f273f737e1b (Nov. 29, 2018… [cited by applicant]
Jing et al., “Visual Search at Pinterest.” Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2015. 10 pages. [cited by applicant]
Kislyuk et al. “Human curation and convnets: Powering item-to-item recommendations on pinterest.” arXiv preprint arXiv:1511.04003 (2015) 6 pages. [cited by applicant]
Kislyuk, “Introducing automatic object detection to visual search,” Pinterest Engineering Blog, https://medium.com/pinterest-engineering/introducing-automatic-object-detection-to-visual-search-e57c29191c30 (Jun. 28, 201… [cited by applicant]
Lee et al., “Efficient object identification and localization for image retrieval using query-by-region.” Computers & Mathematics with Applications 63.2 (2012): 511-517. [cited by applicant]
Microsoft Bing. “Object Detection for Visual Search in Bing.” https://blogs.bing.com/search-quality-insights/2017-09/Object-Detection-in-Visual-Search (Sep. 19, 2017), 7 pages. [cited by applicant]
Yang et al., “Parallel content-based sub-image retrieval using hierarchical searching.” Bioinformatics 30.7 (2014): 996-1002. [cited by applicant]
Gondra. “Personalized content-based image retrieval.” Personalized Information Retrieval and Access: Concepts, Methods and Practices. IGI Global, 2008. 28 pages. [cited by applicant]
International Search Report and Written Opinion mailed Dec. 23, 2022 in PCT/EP2022/075650; 12 pages. [cited by applicant]
Smith et al. “VisualSEEk: a fully automated content-based image query system.” Proceedings of the fourth ACM International conference on Multimedia. 1997. 12 pages. [cited by applicant]
International Search Report Search Report/Office Action mailed Aug. 30, 2023 in P202001501TW01; 8 pages. [cited by applicant]
European Patent Office, “EP Examination Report”, Feb. 12, 2025, 08 pages, EP Application No 22786906.2. [cited by applicant]
Japan Patent Office, “Notice of Reasons for Refusal” Feb. 26, 2026, 09 Pages, JP Application No. 2024-516634. [cited by applicant]
European Patent Office, “Summons to attend oral proceedings pursuant to Rule 115(1) EPC”, Jun. 1, 2026, 11 pages, EP Application No 22786906.2. [cited by applicant]