System and method for enhancing crawling by extracting requests for webpages in an information flow
View Patent ↗A method for providing searching and alerting capabilities in traffic content at access points in data networks is disclosed. Typical access points for Internet, intranet and wireless traffic are described. Traffic flow through an Internet Service Provider is used as a preferred embodiment to exemplify the data traffic used as the input source in the invention. The invention teaches how proper privacy and content filters can be applied to the traffic source. The filtered data stream from the traffic flow can be used to improve the quality of existing searching and alerting services. The invention also teaches how a cache can be developed optimized for holding fresh searchable information captured in the traffic flow. It is further disclosed how the said cache can be converted to a searchable index and either separately or in cooperation with external search indexes be used as a basis for improved search services. The invention also discloses how the traffic flow can be analyzed in order to derive added information for measuring document relevance, access similarity between documents, personalized ranking of search results, and regional differences in document accesses.
1. A method for searching and analysing the traffic content at access points in data networks, wherein the data networks are shared network resources on the Internet, wherein the method comprises:
collecting information in the form of data extracted from the information flow at said access points in the data network,
indexing said collected information,
searching said indexed information,
retrieving information based on the searching,
storing collected information by caching in one or more cache means provided at one or more access points,
processing the cached information repeatedly and generating an index thereof, and
using the traffic content observed between two indexing operations on the cached information for generating a temporal search index with fresh traffic content since a last indexing operation performed of said cached information, and performing searching by combining searching in both said temporal search index and a most recent search index generated by indexing said cached information.
2. A method according to claim 1 , further comprising limiting the indexing of the collected information to the subset of the traffic content that has not been discarded by a sensitivity filter, said sensitivity filter being adapted for detecting one or more of the following, pornographic material, private material, and material from http POST operations.
3. A method according to claim 1 , wherein the searching step includes a substep for relevancy ranking or data filtering depending on a recorded time attribute of traffic content units, said recorded time attribute being one or more of the following-the creation time of a traffic content unit, a last time a traffic content unit is modified, a time of a first observation of a given traffic content unit at an access point, and a time of a last observation of the given traffic content unit at said access point.
4. A method according to claim 1 , wherein the searching step includes a substep for relevancy ranking depending on access counting of individual traffic content units through an access point.
5. A method for searching and analysing the traffic content at access points in data networks, wherein the data networks are shared network resources on the Internet, wherein the method comprises:
collecting information in the form of data extracted from the information flow at said access points in the data network,
indexing said collected information,
searching said indexed information, and
retrieving information based on the searching,
wherein the searching step is implemented by at least one collaborating search engine, the searching step including substeps for dispatching search requests to said at least one collaborating search engine, collecting search result from a local traffic index, collecting search results from said at least one collaborating search engine and combining said collected search results to provide a unified result to an initial search request.
6. A method according to claim 5 , wherein said collaborating search engine is an Internet search engine, and the method further comprises indexing information crawled from the Internet with said search engine.
7. A method according to claim 5 , wherein the searching step is implemented by more than one collaborating search engine, and the method further comprises providing a respective search engine in at least one access point in the data network.
8. A method according to claim 5 , wherein the step for collecting information includes substeps for detecting requested information that was inaccessible, and using information about said inaccessible information to either remove or reduce ranking of an associated entry in a search index.
9. A method according to claim 5 , wherein the searching step is implemented by at least one collaborating search engine, and wherein the step for collecting information includes substeps for detecting new or updated information that has not previously been observed at an access point and using information about the new or updated information to augment a search index of said at least one collaborating search engine.
10. A method for searching and analysing the traffic content at access points in data networks, wherein the data networks are shared network resources on the Internet, wherein the method comprises:
collecting information in the form of data extracted from the information flow at said access points in the data network,
indexing said collected information,
searching said indexed information,
retrieving information based on the searching, and
collecting document identifiers for requested documents, annotating said documents identifiers with spatial information about users submitting the requests, computing access statistics for at least one document including at least the number of document requests from a spatial region and the total number of requests from said spatial region, and determining which documents are most specific for a given spatial region by comparing the access statistics for said given spatial region with the corresponding access statistics for at least a second spatial region.
11. A method according to claim 10 , further comprising deriving said spatial information from user data recorded by an Internet service provider.
12. A method according to claim 10 , further comprising deriving said spatial information from a location of a mobile device requesting a document.
13. A method according to claim 10 , further comprising selecting a spatial granularity such that a sufficient number of users is obtained within every spatial region in order to ensure that individual users cannot be identified in the access statistics.
14. A method according to claim 10 , further comprising determining a document specificity by computing a likelihood that access statistics for at least two spatial regions belongs to a same statistical distribution.
15. A method according to claim 10 , further comprising determining a document specificity by using a large spatial region as the statistics for a null hypothesis and computing a statistical significance of a difference test on access statistics for a given spatial region.
16. A method according to claim 14 , further comprising using a region specific scoring to provide location sensitive ranking of search results.
17. A method according to claim 15 , further comprising using a region specific scoring to provide traffic dependent lists of popular region specific documents.
18. A method according to claim 17 , further comprising dividing said document lists into category sublists by automatic category analysis of the document content.
19. A method for searching and analysing the traffic content at access points in data networks, wherein the data networks are shared network resources on the Internet, wherein the method comprises:
collecting information in the form of data extracted from the information flow at said access points in the data network,
indexing said collected information,
searching said indexed information, and
retrieving information based on the searching,
collecting document identifiers for the requested documents, annotating document requests such that consecutive requests from the same user can be identified, and computing a document similarity between a document “b” and a reference document “a” by comparing the number of “b” requests in the proximity of “a” requests with an average frequency of “b” requests.
20. A method according to claim 19 , further comprising using the document similarity for providing a ranked list of similar documents based on any given document input.
21. A method according to claim 20 , further comprising computing said ranked list by omitting documents from the same site.
22. A method according to claim 19 , further comprising combining the document similarity either with a similarity measurement based on the document content or with a similarity measurement based on counting the number of co-citations.
23. A method according to claim 19 , further comprising computing the document similarity for a group of document identifiers by treating all requests to one of the documents in said group as a request for a single document in the group.
24. A method according to claim 23 , further comprising using a subset of a user's web log as said group of document identifiers.
25. A method according to claim 24 , further comprising taking said group of document identifiers and creating personalized ranking of search results based on said document similarity.
26. A method according to claim 24 , further comprising computing document ranking on the basis of link topology analysis, said document similarity being used to define a link topology and associated weights.