Mobile sitemaps
A method of analyzing documents or relationships between documents includes receiving a notification of an available metadata document containing information about one or more network-accessible documents, obtaining a document format indicator associated with the metadata document, selecting a document crawler using the document format indicator, and crawling at least some of the network-accessible documents using the selected document crawler.
1. A computer-implemented method of analyzing documents or relationships between documents, comprising:
receiving a notification of an available metadata document containing information about one or more network-accessible documents;
obtaining a document format indicator associated with the metadata document, the document format indicator specifying a format in which content of at least one of the network-accessible documents is stored;
selecting, using the document format indicator, a document crawler having an operating mode that defines one or more content formats that the document crawler is capable of accessing, including the format specified by the document format indicator; and
crawling with a computer at least some of the network-accessible documents using the selected document crawler and operating mode.
2. The method of claim 1 , wherein the one or more network-accessible documents comprise a plurality of web pages at a common domain.
3. The method of claim 1 , wherein the metadata document comprises a list of document identifiers.
4. The method of claim 3 , wherein the one or more network-accessible documents comprise a plurality of web pages at a common domain.
5. The method of claim 1 , wherein the document format indicator indicates one or more mobile content formats.
6. The method of claim 5 , wherein the mobile content formats are selected from the group consisting of XHTML, WML, iMode, and HTML.
7. The method of claim 1 , further comprising adding information retrieved from crawling at least some of the network-accessible documents to an index.
8. The method of claim 7 , further comprising receiving a search request from a mobile device and transmitting search results to the mobile device using information in the index.
9. The method of claim 1 , wherein the available metadata document comprises an index referencing a plurality of lists of documents.
10. The method of claim 1 , further comprising receiving an indication of document type for the one or more network-accessible documents and classifying the documents using the indication of document type.
11. The method of claim 10 , further comprising verifying the identity of a provider of the indication of document type to ensure that the provider is trusted.
12. The method of claim 10 , wherein the document type is selected from a group consisting of news, entertainment, commerce, sports, travel, games, and finance.
13. A system for crawling network-accessible documents, comprising:
a memory storing organizational information about network-accessible documents at one or more websites, and format information for the documents;
a crawler configured to access the network-accessible documents using the organizational information; and
a format selector associated with the crawler to cause the crawler to assume a persona compatible with formats indicated by the format information.
14. The system of claim 13 , wherein the organizational information comprises a list of URLs.
15. The system of claim 13 , further comprising an agent repository that stores parameters for causing the crawler to assume a selected persona.
16. A system for crawling network-accessible documents, comprising:
a memory storing organizational information about network-accessible documents at one or more websites, and format information for the documents;
a crawler configured to access the network-accessible documents using the organizational information; and
means for selecting a crawler persona to present in accessing the network-accessible documents.