System and method for automated discovery, binding, and integration of non-registered geospatial web services
View Patent ↗A method and computer system for identifying internet web pages containing documents that comply with a predetermined XML schema. The method includes searching the internet with a search engine for web pages using initial search terms and identifying a first set of HTTP URLs, web crawling at least the first set of HTTP URLs to identify additional HTTP URLs, appending a query to the identified URLs, and evaluating the responses to the query to determine which responses comply with the predetermined XML schema. The XML schema can be a Web Mapping Services schema. The system can store responses that comply with the XML schema in a database of servers, periodically check the database for validity, and convert the map requests for map servers in the database to a GIDB Portal Interface API.
1. A computer-based method for identifying internet web pages containing documents that comply with a predetermined XML schema, comprising:
searching the internet with a search engine for web pages using initial search terms and identifying a first set of HTTP URLs;
web crawling at least the first set of HTTP URLs to identify additional HTTP URLs;
appending a query to the identified URLs; and evaluating the responses to the query to determine which responses comply with the predetermined XML schema; and
storing the HTTP URLs responses that comply with the predetermined XML schema in a database.
2. The method according to claim 1 , further comprising:
the web crawler selecting URLs of the first set of HTTP URLs that are a document type that is unlikely to contain more HTTP URLs, and not web crawling the selected URLs.
3. The method according to claim 2 , wherein the selected URLs are at least one of music files, image files, and executable files.
4. The method according to claim 1 , further comprising:
comparing a newly discovered document against previously known WMS server documents to avoid duplication.
5. The method according to claim 1 , further comprising:
adding the HTTP URLs responses that comply with the predetermined XML schema to a database of available servers.
6. The method according to claim 4 , further comprising web crawling the HTTP URLS in the database periodically to discover non-operational links.
7. The method according to claim 1 , wherein the web crawling includes web crawling a predetermined set of additional web sites.
8. The method according to claim 1 , wherein predetermined set of additional web sites include GIS web sites.
9. The method according to claim 1 , wherein the predetermined XML schema is a schema published by the Open Geospatial Consortium.
10. The method according to claim 1 , wherein the appended query requests a Web Mapping Services Capabilities document.
11. The method according to claim 1 , wherein the appended query includes ?REQUEST=GetCapabilities.
12. The method according to claim 1 , wherein the appended query includes ?REQUEST=GetCapabilities&SERVICE=WMS.
13. The method according to claim 1 , wherein said searching includes searching using an API.
14. The method according to claim 1 , further comprising:
repeating said searching, said web crawling, said appending and said evaluating at periodic intervals.
15. The method according to claim 5 , further comprising:
converting map requests for at least one particular map server in the database to a GIDB Portal Interface API.
16. A computer system including a server connected to the internet and having software for identifying internet web pages containing documents that comply with a predetermined XML schema, the software configured to:
search the internet with a search engine for web pages using initial search terms and identify a first set of HTTP URLs;
web crawl at least the first set of HTTP URLs to identify additional HTTP URLs;
append a query to the identified URLs;
evaluate the responses to the query to determine which responses comply with the predetermined XML schema; and
store a list of valued URLs having responses with the predetermined XML schema.
17. The system according to claim 16 , further comprising:
the web crawler selecting URLs of the first set of HTTP URLs that are a document type that is unlikely to contain more HTTP URLs, and not web crawling the selected URLs.
18. The system according to claim 17 , wherein the selected URLs are at least one of music files, image files, and executable files.
19. The system according to claim 16 , further comprising:
comparing a newly discovered document against previously known WMS server documents to avoid duplication.
20. The system according to claim 16 , further comprising:
adding the HTTP URLs responses that comply with the predetermined XML schema to a database of available servers.
21. The system according to claim 19 , further comprising web crawling the HTTP URLS in the database periodically to discover non-operational links.
22. The system according to claim 16 , wherein the web crawling includes web crawling a predetermined set of additional web sites.
23. The system according to claim 17 , wherein predetermined set of additional web sites include GIS web sites.
24. The system according to claim 17 , wherein the predetermined XML schema is a schema published by the Open Geospatial Consortium.
25. The system according to claim 17 , wherein the appended query requests a Web Mapping Services Capabilities document.
26. The system according to claim 16 , wherein the appended query includes ?REQUEST=GetCapabilities.
27. The system according to claim 16 , wherein the appended query includes ?REQUEST=GetCapabilities&SERVICE=WMS.
28. The system according to claim 16 , wherein said searching includes searching using an API.
29. The system according to claim 16 , further comprising:
repeating said searching, said web crawling, said appending and said evaluating at periodic intervals.
30. The system according to claim 16 , further comprising:
converting map requests for at least one particular map server in the database to a GIDB Portal Interface API.