IP Library › Granted Patent US 12,488,005
Granted Patent B1
US 12,488,005 · App. 16/732,119 · Granted Dec 2, 2025

Query to interest mapping

Inventors: Jinfeng Zhuang (Sammamish, WA); Jinyu Xie (San Mateo, CA); Yunsong Guo (Santa Clara, CA)
Assignee: Pinterest, Inc.
G06F16/2457G06F16/2423G06F16/248G06F16/285G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,488,005
App. No.
16/732,119
Granted
Dec 2, 2025
Kind
B1
Abstract

Systems and methods for identifying relevant content within a corpus of visual content items in response to a user's text-based query are presented. In response to a text-based query, the query is mapped to a most-engaged content item of the corpus of visual content items included in responses to the query from a plurality of users. At least one text-based term associated with the most-engaged content item is identified and combined with the query from an expanded query. The expanded query is mapped to an interest node of an interest taxonomy and content items associated with the mapped interest node are identified. At least some of the content items associated with the mapped interest node are selected and returned as response content to the received query.

Claims (82)

1 . A computer-implemented method for providing a content response to a query from a user, the computer-implemented method comprising:

receiving a text-based query from a user;

determining a visual content item from a corpus of visual content items based at least in part on a frequency that the visual content item is included in query responses to the received text-based query from a plurality of users;

determining at least one text-based term associated with the visual content item that is not already included in the received text-based query;

generating an expanded query by combining the at least one text-based term associated with the visual content item with the received text-based query;

mapping the expanded query to an interest node of an interest taxonomy;

identifying a set of visual content items associated with the mapped interest node of the interest taxonomy;

selecting a subset of visual content items from the set of visual content items associated with the mapped interest node of the interest taxonomy; and

providing the subset of visual content items as a response to the received text-based query.

2 . The computer-implemented method of claim 1 , further comprising:

identifying a plurality of text-based terms associated with the visual content item; and

wherein determining the at least one text-based term associated with the visual content item that is not already included in the received text-based query comprises determining at least one text-based term from the plurality of text-based terms associated with the visual content item.

3 . The computer-implemented method of claim 2 , further comprising:

selecting multiple text-based terms of the plurality of text-based terms associated with the visual content item including the at least one text-based term; and

generating the expanded query by combining the multiple text-based terms associated with the visual content item with the received text-based query.

4 . The computer-implemented method of claim 1 , further comprising:

determining a plurality of visual content items from the corpus of visual content items based at least in part on a second frequency that the plurality of visual content items is included in query responses to the received text-based query, including the visual content item;

determining a plurality of text-based terms associated with the plurality of visual content items, including the at least one text-based term; and

generating a second expanded query by combining the plurality of text-based terms associated with the plurality of visual content items with the received text-based query.

5 . The computer-implemented method of claim 4 , further comprising:

mapping the second expanded query to a plurality of interest nodes of the interest taxonomy, including the interest node, using a trained mapping model; and

identifying a second set of visual content items associated with the plurality of mapped interest nodes of the interest taxonomy.

6 . The computer-implemented method of claim 5 , wherein mapping the expanded query to the plurality of interest nodes of the interest taxonomy using the trained mapping model comprises:

associating a score with interest nodes of the interest taxonomy using the trained mapping model, each score indicative of an associated interest node being a match for the expanded query; and

identifying the plurality of interest nodes of the interest taxonomy having the highest associated scores.

7 . The computer-implemented method of claim 1 , wherein determining the at least one text-based term associated with the visual content item that is not already included in the received text-based query comprises:

identifying textual content associated with the visual content item from any one or more of a user's annotations of the visual content item, a file name associated with the visual content item, and a source path indicative of a source location of an external source of the visual content item; and

determining the at least one text-based term from the identified textual content.

8 . The computer-implemented method of claim 1 , wherein determining the at least one text-based term associated with the visual content item not already included in the received text-based query comprises:

processing the visual content item by a content classifier to generate text-based classification information for the visual content item; and

determining the at least one text-based term from the classification information generated by the content classifier.

9 . A computer-readable medium bearing computer-executable instructions which, when executed on a computing system comprising at least a processor executing the instructions, carry out a method for providing a response to a text-based query, the method comprising:

receiving a text-based query from a user;

determining a visual content item from a corpus of visual content items based at least in part on a frequency that the visual content item is included in query responses to the received text-based query from a plurality of users;

identifying textual content associated with the visual content item;

determining a plurality of text-based terms from the textual content associated with the visual content item;

generating an expanded query by combining the plurality of text-based terms associated with the visual content item with the received text-based query;

mapping the expanded query to an interest node of an interest taxonomy;

identifying a set of visual content items associated with the mapped interest node of the interest taxonomy;

selecting a subset of visual content items from the set of visual content items associated with the mapped interest node of the interest taxonomy; and

providing the subset of visual content items as a response to the received text-based query.

10 . The computer-readable medium of claim 9 , the method further comprising:

mapping the expanded query to a plurality of interest nodes of the interest taxonomy, the plurality of interest nodes including the mapped interest node; and

identifying a second set of visual content items associated with the plurality of mapped interest nodes of the interest taxonomy, including the mapped interest node.

11 . The computer-readable medium of claim 10 , wherein mapping the expanded query to the plurality of interest nodes of the interest taxonomy comprises:

associating a score with interest nodes of the interest taxonomy, each score indicative of an associated interest node being a match for the expanded query; and

identifying the plurality of interest nodes of the interest taxonomy having the highest associated scores.

12 . The computer-readable medium of claim 9 , the method further comprising:

determining a plurality of visual content items from the corpus of visual content items based at least in part on a second frequency that the plurality of visual content items are included in query responses to the received text-based query;

identifying textual content associated with the plurality of visual content items;

determining a plurality of text-based terms associated with the plurality of visual content items from the identified textual content associated with the plurality of visual content items that are not already included in the received text-based query; and

generating a second expanded query by combining the plurality of text-based terms associated with the plurality of visual content items with the received text-based query.

13 . The computer-readable medium of claim 11 , wherein:

each score associated with the associated interest node of the interest taxonomy is a predicted score indicative of the associated interest node being a match for the expanded query; and

the predicted scores associated with the interest nodes of the interest taxonomy are generated according to a trained deep neural network.

14 . The computer-readable medium of claim 13 , wherein the trained deep neural network comprises any one of a trained fastText neural network, a trained convolutional neural network (CNN), a trained recurrent neural network (RNN), a trained CNN+RNN vertical neural network, a trained CNN+RNN horizontal neural network, or a trained hierarchical attention network (HAN).

15 . The computer-readable medium of claim 14 , the method further comprising:

conducting an evaluation of the mapping of the expanded query to the interest node of the interest taxonomy and the content items of the response content;

retraining the trained deep neural network with the results of the evaluation; and

associating the interest node to the text-based query as a likely interest node for the text-based query.

16 . The computer-readable medium of claim 9 , wherein identifying textual content associated with the visual content item comprises, upon a determination that no textual content is associated with the visual content item:

classifying the visual content item using a content classifier resulting in text-based classification information; and

utilizing the text-based classification information as the textual content associated with the visual content item.

17 . A computing system for responding to a text-based query from a computer user, the computing system comprising a processor and a memory, wherein the processor executes instructions stored in the memory as part of or in conjunction with additional components to respond to the text-based query, and configures the computing system to at least:

receive a text-based query from a user;

determine a visual content item from a corpus of visual content items based at least in part on a frequency that the visual content item is included in a plurality of query responses to the received text-based query from a plurality of users;

identify textual content associated with the visual content item;

determine a plurality of text-based terms from the textual content associated with the visual content item;

generate an expanded query by combining the plurality of text-based terms associated with the visual content item with the received text-based query;

map the expanded query to an interest node of an interest taxonomy using a trained mapping model;

identify a set of visual content items associated with the mapped interest node of the interest taxonomy;

select a subset of visual content items from the set of visual content items; and

provide the subset of visual content items as a response to the received text-based query.

18 . The computing system of claim 17 , wherein the computing system is further configured to at least:

map the expanded query to a plurality of interest nodes of the interest taxonomy using the trained mapping model, the plurality of interest nodes including the interest node; and

identify a second set of visual content items associated with the plurality of mapped interest nodes of the interest taxonomy, including the mapped interest node.

19 . The computing system of claim 18 , wherein in mapping the expanded query to the plurality of interest nodes of the interest taxonomy using the trained mapping model, the computer system is further configured to:

associate a score with interest nodes of the interest taxonomy using the trained mapping model, each score indicative of the associated interest node being a match for the expanded query; and

identify the plurality of interest nodes of the interest taxonomy as having the highest associated scores.

20 . The computing system of claim 17 , wherein in identifying textual content associated with the visual content item, upon a determination that no textual content is associated with the visual content item, the computer system is configured to:

classify the visual content item using a content classifier resulting in text-based classification information; and

utilize the text-based classification information as the textual content associated with the visual content item.

Assignments (2)
SECURITY INTEREST Recorded Oct 25, 2022
From: PINTEREST, INC.
To: JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT
Reel/Frame 061767/0853 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 31, 2019
From: ZHUANG, JINFENG; XIE, JINYU; GUO, YUNSONG
To: PINTEREST, INC.
Reel/Frame 051395/0477 →
Continuity (1)
Provisional Application 62909134 · Oct 1, 2019
References Cited (43)
US 9015185B2 · Kanigsberg · 2015 [cited by examiner]
US 9767204B1 · Hoover · 2017 [cited by examiner]
US 20080250011A1 · Haubold · 2008 [cited by examiner]
US 20150178392A1 · Jockisch · 2015 [cited by examiner]
US 20200302331A1 · Lonij · 2020 [cited by examiner]
Bojanowski, P. et al., 2017, Enriching Word Vectors with Subword Information. TACL 5 (2017), pp. 135-146, Retrieved: https://www.mitpressjournals.org/doi/pdfplus/10.1162/tacl_a_00051, 13 pages. [cited by applicant]
Cer, D. et al., 2018, Universal Sentence Encoder. CoRR abs/1803.11175 (2018), arXiv:1803.11175, Retrieved: https://arxiv.org/pdf/1803.11175.pdf, 7 pages. [cited by applicant]
Chen, T. and Guestrin, C. 2016. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, Aug. 13-17, 2016, … [cited by applicant]
Cho, K. et al., 2014, Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014,… [cited by applicant]
Conneau, A. et al., 2017, Very Deep Convolutional Networks for Text Classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Sp… [cited by applicant]
Cortes, C. and Vapnik, V., 1995, Support-Vector Networks. Machine Learning 20, 3 (1995), pp. 273-297, Retrieved: http://image.diku.dk/imagecanon/material/cortes_vapnik95.pdf, 25 pages. [cited by applicant]
Dai, Z. et al., 2019, Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, Jul. 28-A… [cited by applicant]
Devlin, J. et al., 2018, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR abs/1810.04805 (2018), arXiv:1810.04805, Retrieved: https://arxiv.org/pdf/1810.04805v1.pdf, 14 pages. [cited by applicant]
Ganti, V. et al., 2010, Precomputing Search Features for Fast and Accurate Query Classification. In Proceedings of the Third International Conference on Web Search and Web Data Mining, WSDM 2010, New York, Ny, USA, Feb.… [cited by applicant]
Hochreiter, S. and Schmidhuber, J., 1997. Long Short-Term Memory. Neural Computation 9, 8 (1997), pp. 1735-1780, Retrieved: http://www.bioinf.jku.at/publications/older/2604.pdf, 32 pages. [cited by applicant]
Howard, J. and Ruder, S., 2018, Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, J… [cited by applicant]
Jansen, B. J. et al., 2008, Determining the Informational, Navigational, and Transactional Intent of Web Queries. Inf. Process. Manage. 44, 3 (2008), pp. 1251-1266, Retrieved: https://faculty.ist.psu.edu/jansen/academic… [cited by applicant]
Joulin, A. et al., 2017, Bag of Tricks for Efficient Text Classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, Apr. … [cited by applicant]
Kim, Y., 2014, Convolutional Neural Networks for Sentence Classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 1746-1751, Retrieved: http://emnlp2014.org/papers/… [cited by applicant]
Kingma, D. P. and Ba, J., 2015, Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. http:/… [cited by applicant]
Kumar, A. et al., 2016, Ask Me Anything: Dynamic Memory Networks for Natural Language Processing. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, Jun. 19-24, 2… [cited by applicant]
Laclavik, M. et al., 2015, Search Query Categorization at Scale. In Proceedings of the 24th International Conference on World Wide Web Companion, WWW 2015, Florence, Italy, May 18-22, 2015—Companion Volume pp. 1281-1286… [cited by applicant]
Lai, S. et al., 2015, Recurrent Convolutional Neural Networks for Text Classification, In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Jan. 25-30, 2015, Austin, Texas, USA, pp. 2267-2273, … [cited by applicant]
Liu, X. et al., 2019, Multi-Task Deep Neural Networks for Natural Language Understanding. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-Aug. 2… [cited by applicant]
Liu, Y. et al., 2019, ROBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR abs/1907.11692 (2019). arXiv:1907.11692 http://arxiv.org/abs/1907.11692, Retrieved: https://arxiv.org/pdf/1907.11692.pdf, 13 pages. [cited by applicant]
Mikolov, T. et al., 2013, Efficient Estimation of Word Representations in Vector Space. CoRR abs/1301.3781 (2013), arXiv:1301.3781 http://arxiv.org/abs/1301.3781, Retrieved: https://arxiv.org/pdf/1301.3781.pdf. [cited by applicant]
Palangi, H., 2016, Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval. IEEE/ACM Trans. Audio, Speech and Language Processing 24, 4 (2016), pp. 694-707, Retri… [cited by applicant]
Peters, M. E. et al., 2018, Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NA… [cited by applicant]
Radford, A. et al., 2018, Improving Language Understanding by Generative Pre-Training. In arxiv, Retrieved: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_p… [cited by applicant]
Radford, A. et al., 2018, Language Models are Unsupervised Multitask Learners, (2018) Retrieved: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf, 24 pag… [cited by applicant]
Sanh, V. et al., 2019, Smaller, faster, cheaper, lighter: Introducing DistilBERT, a distilled version of BERT, Retrieved: https://medium.com/huggingface/distilbert-8cf3380435b5, 11 pages. [cited by applicant]
Speer, R. et al., 2017, ConceptNet 5.5: An Open Multilingual Graph of General Knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, Feb. 4-9, 2017, San Francisco, California, USA, pp.… [cited by applicant]
Sriram, B. et al., 2010, Short Text Classification in Twitter To Improve Information Filtering. In Proceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 20… [cited by applicant]
Sun, C. et al., 2019, How to Fine-Tune BERT for Text Classification? CoRR abs/1905.05583 (2019). arXiv: 1905.05583 http://arxiv.org/abs/1905.05583, Retrieved: https://arxiv.org/pdf/1905.05583.pdf, 10 pages. [cited by applicant]
Vaswani, A. et al., 2017, Attention is All you Need. In Annual Conference on Neural Information Processing Systems 2017, Dec. 4-9, 2017, Long Beach, Ca, USA, pp. 6000-6010, Retrieved: https://arxiv.org/pdf/1706.03762.pd… [cited by applicant]
Vinicombe, H., 2019, Understanding Pins Through Key-Word Extraction, Retrieved: https://medium.com/pinterest-engineering/understanding-pins-through-keyword-extraction-40cf94214c18, 9 pages. [cited by applicant]
Wang, J. et al., 2017, Combining Knowledge with Deep Convolutional Neural Networks for Short Text Classification. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017,… [cited by applicant]
Wang, Z. et al., 2015, Query Understanding through Knowledge-Based Conceptualization. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, J… [cited by applicant]
Yang, Z. et al., 2016, Hierarchical Attention Networks for Document Classification. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T… [cited by applicant]
Yang, Z. et al., 2019, XLNet: Generalized Autoregressive Pretraining for Language Understanding. CoRR abs/1906.08237 (2019), Retrieved: https://arxiv.org/pdf/1906.08237.pdf, 18 pages. [cited by applicant]
Zhai, Z. et al., 2019, Learning a Unified Embedding for Visual Search at Pinterest. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, Aug.… [cited by applicant]
Zhang, Y. and Wallace, B. C., 2017, A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification. In Proceedings of the Eighth International Joint Conference on Natur… [cited by applicant]
Zhuang J. and Liu, Y., 2019, Pin Text: A Multitask Text Embedding System in Pinterest. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, A… [cited by applicant]