Indexing system and method

1. An indexing system comprising:

a plurality of server based agents corresponding to a plurality of servers, each of the servers providing access to and serving data for at least one site on the server;

each of the server based agents being associated with a corresponding server and forming an index file of site data on the agent's corresponding server and representing changes to the site data determined by the agent on the agent's corresponding server; and

a central index that stores index information formed from the index files received from the plurality of server based agents;

each of the server based agents initiating communication over the Internet with the central index to thereby transfer the agent's index file for the agent's corresponding server to the central index, the central index being used by a search engine;

at least one of the sites serving at least one dynamic page;

the index file associated with the dynamic page including inputs that will generate a valid page by a corresponding form handling program; and

each of the server based agents associated with a dynamic page identifying at least one database used to create the dynamic page, and introducing a trigger in the database to identify any changes in the database, the changes being included in an update index file associated with the dynamic page.

2. The indexing system as claimed in claim 1 , wherein each index file includes information on any change affecting the validity of links.

3. The indexing system as claimed in claim 2 , wherein each of the server based agents, upon identifying said link validity change, is arranged to transmit notification of the change to a corresponding server associated with another site that includes an invalid link resulting from the change.

4. The indexing system as claimed in claim 3 , wherein the central index is adapted to identify a site affected by the link validity change and to transmit notification to a corresponding server associated with the affected site.

5. The indexing system as claimed in claim 3 or 4 , wherein the notification is transmitted to the respective server based agent of the corresponding server which is adapted to effect action to update information on the corresponding server.

6. The indexing system as claimed in claim 3 or 4 , wherein the notification is transmitted to an administrator of the corresponding server and, upon receipt of the notification, the administrator is able to effect action to update information on the corresponding server.

7. The indexing system as claimed in claim 1 , wherein the index file associated with the dynamic page further includes outputs and text of the dynamic page.

8. The indexing system as claimed in claim 1 , wherein the server based agent associated with the dynamic page is adapted to access the database and include one or more columns of the database in the index file associated with the dynamic page.

9. The indexing system as claimed in claim 1 , wherein the index file associated with the dynamic page includes a record of inputs of pages previously retrieved therefrom by users.

10. The indexing system as claimed in claim 1 , wherein the index file associated with the dynamic page includes an index of the full text of the dynamic page, previously retrieved by users.

11. The indexing system as claimed in claim 1 , wherein the index file associated with each dynamic page for which input/output tuples or row/column indexing is stored further includes an identifier for identifying the form handling program and text associated with the page.

12. The indexing system as claimed in claim 1 , including a server for accessing the central index in response to search queries.

13. The indexing system as claimed in claim 1 , wherein each said server based agent initiates said communication by storing a file in a predetermined location.

14. A method of indexing comprising the steps of:

providing a plurality of server based agents corresponding to a plurality of servers, each server based agent being associated with a corresponding server for indexing sites located on the corresponding server that are served by said server;

having each server based agent form an index file of site data of the sites on said agent's corresponding server and representing changes to the site data determined by the agent on the agent's corresponding server;

providing a central index that stores index information compiled from the index files received from the plurality of server based agents;

having each of the server based agents initiate communication over the Internet with the central index to thereby transfer the index file for the agent's corresponding server from the server based agent to the central index, the central index being usable by a search engine,

the index file including information relating to dynamic pages of the sites,

each of the server based agents associated with a dynamic page identifying at least one database used to create the dynamic page, and extracting the text and input tuples for inclusion in the index file; and introducing a trigger in the database to report to the server based agent, when changes occur in the database, the changes being included in an update index file associated with the dynamic page.

15. The method of indexing as claimed in claim 14 , wherein each server based agent records link information on the sites and any change affecting the validity of the link information is recorded in the index file formed by the server based agent.

16. The method of indexing as claimed in claim 15 , including the server based agent identifying said link validity change and transmitting notification of the change to a corresponding server associated with another site that includes an invalid link resulting from the change.

17. The method of indexing as claimed in claim 15 , wherein the central index has a list of site addresses and, upon receipt of an index file indicating said change, an associated site affected by the change is identified and notification is sent to the corresponding server of the affected site.

18. The method of indexing as claimed in claim 16 or 17 , wherein the notification is transmitted to the respective server agent of the corresponding server which effects action to update information on the corresponding server.

19. The method of indexing as claimed in claim 16 or 17 , wherein the notification is transmitted to an administrator of the corresponding server and, upon receipt of the notification, the administrator is able to effect action to update information on the corresponding server.

20. The method of indexing as claimed in claim 14 , wherein the index file includes inputs to generate the dynamic pages.

21. The method of indexing as claimed in claim 20 , wherein an index file includes outputs used to generate the dynamic pages.

22. The method of indexing as claimed in claim 14 , wherein the server based agent extracts output tuples associated with said input tuples.

23. The method of indexing as claimed in 22 , wherein said server based agent extracts columns of the database for inclusion in the index file.

24. The method of indexing as claimed in claim 14 , wherein the index file includes a record of data used to generate previously retrieved dynamic pages.

25. The method of indexing as claimed in claim 14 , wherein the index file includes an index of the text of dynamic pages previously retrieved by users.

26. The method as claimed in claim 14 , wherein each said server agent initiates said communication by storing a file in a predetermined location.