Discovering relevant concept and context for content node
Discovering relevant concepts and context for content nodes to determine a user's intent includes identifying one or more concept candidates in a content node based at least in part on one or more statistical measures, and matching concepts in a concept association map against text in the content node. The concept association map represents concepts, concept metadata, and relationships between the concepts. The one or more concept candidates are ranked to create a ranked one or more concept candidates based at least in part on a measure of relevance. The ranked one or more concept candidates is expanded according to one or more cost functions. The expanded set of concepts is stored in association with the content node.
1. A computerized method comprising:
extracting, by a candidate concept extractor, one or more concept candidates from a content node with a computer based at least in part on:
one or more statistical measures, and
matching concepts in a concept association map against text in the content node, the concept association map representing concepts, concept metadata, and relationships between the concepts, wherein such a map is dynamic and constantly updated, wherein the results on the concept association map are clustered, wherein the content nodes are tagged with labels representing at least one high level category and further wherein the concept association map is augmented by adding links between at least one of a search query and the concepts;
ranking, by a concept filterer, the one or more concept candidates to create a ranked one or more concept candidates based at least in part on a measure of relevance with the computer after extracting the one or more concepts in the content node;
expanding, by a concept expander, the ranked one or more concept candidates according to one or more cost functions, the expanding creating an expanded set of concepts with the computer after ranking the one or more concept candidates; and
storing, in a memory, the expanded set of concepts in association with the content node after expanding the ranked one or more concept candidates.
2. The method of claim 1 , wherein the one or more statistical measures comprises one or more of:
an indication of a likelihood that a given n-gram will appear in a document that is part of a corpus;
a frequency of n-grams in the content node;
a similarity of the content node to other content nodes for which relevant concept candidates have already been identified; and
a weight of the content node in the concept association map.
3. The method of claim 1 , wherein the measure of relevance weighs:
a frequency that the one or more concept candidates occurs in the content node;
a likelihood of the one or more concept candidates appearing in a document; and
a likelihood of the one or more concept candidates being selected based on a closeness of the content node to similar concept nodes.
4. The method of claim 1 , further comprising:
before the identifying, classifying the content of the content node.
5. The method of claim 1 , wherein the concept association map is derived from one or more of:
concept relationships found on the World Wide Web;
associations derived from user browsing history;
advertisers bidding campaigns;
taxonomies; and
encyclopedias.
6. The method of claim 1 , wherein the content node comprises a user query having a set of search keywords.
7. The method of claim 1 , wherein the one or more concept candidates are provided by a user.
8. The method of claim 1 , wherein the one or more cost functions comprises selecting neighbor nodes that are more likely to be strongly associated with each other.
9. The method of claim 1 , wherein the one or more cost functions comprises selecting neighbor nodes that, based on aggregate user activity history, have a higher likelihood to be associated.
10. The method of claim 1 , wherein the one or more cost functions comprises determining which neighboring concepts in the concept association map are tied to a form of monetization that yields a highest conversion rate.
11. The method of claim 1 , further comprising mapping one or more advertisements to the content node based at least in part on the expanded set of concepts.
12. An apparatus comprising:
memory comprising a concept association map representing concepts, concept metadata, and relationships between the concepts, wherein such a map is dynamic and constantly updated, wherein the results on the concept association map are clustered, wherein the content nodes are tagged with labels representing at least one high level category and further wherein the concept association map is augmented by adding links between at least one of a search query and the concepts; and
a processor comprising:
a candidate concept extractor configured to extract one or more concept candidates from a content node based at least in part on:
one or more statistical measures, and
matching concepts in a concept association map against text in the content node;
a concept filterer configured to rank the one or more concept candidates to create a ranked one or more concept candidates based at least in part on a measure of relevance after the candidate concept extractor extracts the one or more concept candidates in the content node; and
a concept expander configured to expand the ranked one or more concept candidates according to one or more cost functions, the expanding creating an expanded set of concepts after the concept filterer ranks the one or more concept candidates, wherein
the apparatus is further configured to store the expanded set of concepts in association with the content node in the memory.
13. The apparatus of claim 12 , wherein the one or more statistical measures comprises one or more of:
an indication of a likelihood that a given n-gram will appear in a document that is part of a corpus;
a frequency of n-grams in the content node;
a similarity of the content node to other content nodes for which relevant concept candidates have already been identified; and
a weight of the content node in the concept association map.
14. The apparatus of claim 12 , wherein the measure of relevance weighs:
a frequency that the one or more concept candidates occurs in the content node;
a likelihood of the one or more concept candidates appearing in a document; and
a likelihood of the one or more concept candidates being selected based on a closeness of the content node to similar concept nodes.
15. The apparatus of claim 12 , wherein the apparatus is further configured to, before the identifying, classify the content of the content node.
16. The apparatus of claim 12 , wherein the concept association map is derived from one or more of:
concept relationships found on the World Wide Web;
associations derived from user browsing history;
advertisers bidding campaigns;
taxonomies; and
encyclopedias.
17. The apparatus of claim 12 , wherein the content node comprises a user query having a set of search keywords.
18. The apparatus of claim 12 , wherein the one or more concept candidates are provided by a user.
19. The apparatus of claim 12 , wherein the concept expander is further configured to select neighbor nodes that are more likely to be strongly associated with each other.
20. The apparatus of claim 12 , wherein the concept expander is further configured to select neighbor nodes that, based on aggregate user activity history, have a higher likelihood to be associated.
21. The apparatus of claim 12 , wherein the concept expander is further configured to determine which neighboring concepts in the concept association map are tied to a form of monetization that yields a highest conversion rate.
22. The apparatus of claim 12 , wherein the apparatus is further configured to map one or more advertisements to the content node based at least in part on the expanded set of concepts.