Automated keyword analysis system and method
An automatic keyword analysis for content illumination and document fragmentation for internet search queries. Initially, the invention obtains keywords from a document. The taxonomy then expands the obtained words from the document. The keywords are further refined using keyword demand data. The keyword demand data includes words where too few or too many searches would hamper the user's search. After refinement, the keyword final set is placed onto the appropriate web page. Consequently, internet search engines “hit” the web page because the automatic keyword analysis added optimal and efficient keywords.
1. A computerized method of automatically identifying keywords relevant to a document invisible to search engines comprising:
analyzing at a computer the document invisible to search engine crawlers to obtain a keyword starter set from the document, the keyword starter set obtained by:
(1) applying at said computer an automated parser to the document to obtain keywords; and
(2) applying a frequency prominence analysis to the keywords to select one or more frequently occurring keywords to add to the keyword starter set;
expanding at the computer the keyword starter set by applying a computerized taxonomy to the keyword starter set to form a keyword super set;
applying at the computer a keyword stop list to keywords in the keyword super set to remove keywords included in the keyword stop list;
refining at the computer the keyword super set to form a keyword final set by applying keyword demand data to the keyword super set to remove one or more additional keywords from the keyword super set, wherein the demand data reflects the frequency of use of the keywords as search terms in internet search engines;
adding at the computer the keyword final set to a web page for accessing the document;
storing the document invisible to search engines for retrieval via the web page for accessing the document;
adding the web page with the keyword final set to a web site to facilitate location by internet search engines of the web page for accessing the document according to the keywords added to the web page; and
providing internet users with access via the web page to the document invisible to search engines.
2. The method of claim 1 , wherein the keyword starter set comprises a plurality of keywords in a document title of the document.
3. The method of claim 1 , wherein the keyword starter set comprises a plurality of keywords in a section title of the document.
4. The method of claim 1 , wherein the keyword starter set comprises a plurality of user provided words.
5. The method of claim 1 wherein obtaining a keyword starter set from the document comprises selecting a batch processor for batch processing the document to produce a keyword starter set.
6. The method of claim 1 further comprising storing the keyword starter set for debugging purposes.
7. The method of claim 1 , wherein expanding the keyword starter set by applying a computerized taxonomy comprises adding a plurality of related keywords.
8. The method of claim 1 , wherein the keyword stop list comprises a maintained list of keywords.
9. The method of claim 1 further comprising storing the keyword super set for debugging purposes.
10. The method of claim 1 , wherein refining the keyword super set to form the keyword final set by applying keyword demand data that reflects the frequency of use of the keywords as search terms in internet search engines further comprises:
removing the search keyword from the keyword super set if the number of successful searches indicated by the keyword demand data is low; and
removing the search keyword from the keyword super set if the number of web pages relevant to the keyword as indicated by the keyword demand data is high.
11. A system for identifying keywords relevant to a search query to add to a web page for accessing a document invisible to search engines comprising:
a server connected to a network, the server uploading a document from a user via the network, the server comprising:
at least one processor;
a database storing a plurality of documents invisible to search engines; and
a memory operatively coupled to the processor, the memory storing program instructions that when executed by the processor, cause the processor to:
generate a keyword starter set from the document invisible to search engines, the keyword starter set generated by:
(1) applying a parser to the document to obtain keywords; and
(2) applying a frequency prominence analysis to the keywords to select frequently occurring keywords to add to the keyword starter set,
add a plurality of related keywords to the keyword starter set by applying a taxonomy to the keyword starter set,
form a keyword super set by eliminating a plurality of stop keywords from the keyword starter set and the plurality of related keywords,
use keyword demand data to remove one or more keywords from the keyword super set to form a keyword final set wherein the demand data indicates the frequency of use of the keywords as search terms in internet search engines,
add the keyword final set to a web page for accessing the document;
store the document in the database of documents invisible to search engines;
add the web page to a web site to facilitate internet search engines indexing of the web page for accessing the document according to the keywords added to the web page; and
provide internet users with access via the web page to the document invisible to search engines.
12. The system of claim 11 , wherein the server uses keyword demand data to remove one or more keywords from the keyword super set to form a keyword final set by:
removing the search keyword from the keyword super set if the number of successful searches indicated by the keyword demand data is low; and
removing the search keyword from the keyword super set if the number of web pages relevant to the keyword as indicated by the keyword demand data is high.
13. A computerized method of identifying keywords relevant to a search query to facilitate search engine location of a web page for accessing a document invisible to search engines comprising:
obtaining at a computer a keyword starter set from the document invisible to search engines;
applying at said computer a computerized taxonomy to the keyword starter set to form a keyword super set by expanding the keyword starter set to include additional words conceptually related to the keywords in the starter set;
applying at the computer keyword demand data to the keyword super set to remove one or more words from the keyword super set to create a keyword final set, wherein the demand data reflects the frequency of use of the words as search terms in internet search engines;
adding at the computer the keyword final set to a web page for accessing the document;
storing the document invisible to search engines for retrieval via the web page for accessing the document;
adding the web page to a web site to facilitate location by internet search engines of the web page for accessing the document according to the keywords from the keyword final set; and
providing internet users with access via the web page to the document invisible to search engines.
14. The method of claim 13 wherein the keyword starter set is obtained by using words from the document title.
15. The method of claim 13 wherein the keyword starter set is obtained by using words from the document title and the section titles of the document.
16. The method of claim 13 wherein the keyword starter set is obtained from a computer user manually entering the keywords in a computer.
17. The method of claim 13 wherein the keyword starter set is obtained automatically by:
a. applying a parser to the document to select keywords based on the word size, location of the word in a sentence, and commonality of the word in the document;
b. applying a frequency prominence analysis to the keywords identified in step (a) to select words that occur frequently in the document.