Distributed content verification and indexing

1. A method of verifying content, the method including the steps of:

receiving, at an indexing system, a first copy of a webpage from a hosting server;

processing the first copy, at the indexing system, to obtain first indicating data including one or more properties of the first copy;

receiving second indicating data from a first client terminal, the second indicating data obtained by processing, at the first client terminal, a second copy of the webpage, the second copy received by the first client terminal from the hosting server, the second indicating data including one or more properties of the second copy;

comparing, at the indexing system, the first indicating data with the second indicating data;

determining that the first copy is verified when a number of non-matching properties between the first indicating data and the second indicating data does not exceed a threshold number;

determining that the first copy is not verified when the number of non-matching properties between the first indicating data and the second indicating data exceeds a threshold number;

upon determining that the first copy is not verified, receiving third indicating data from a second client terminal, the third indicating data obtained by processing, at the second client terminal, a third copy of the webpage, the third copy received at the second client terminal from the hosting server, the third indicating data including one or more properties of the third copy of the webpage;

determining whether the first copy of the webpage received by the indexing system is verified or not verified based on a comparison between the first indicating data and the third indicating data; and

upon determining that the first copy is verified, indexing content of the webpage.

2. The method as claimed in claim 1 , wherein the content is webpage content.

3. The method as claimed in claim 1 , wherein the comparison involves determining the number of matching properties between the first indicating data and the second indicating data.

4. The method as claimed in claim 1 , wherein if the first copy is determined to be verified, the contents of the first copy are indexed.

5. The method as claimed in claim 1 , wherein the first indicating data includes an index of the contents of the first copy, and if the first copy is determined to be verified, then the index is considered verified.

6. The method as claimed in claim 1 , wherein the first indicating data or the second indicating data are at least partially obtained by one or more of: performing natural language analysis; performing word frequency analysis; performing a word hash; performing a word group hash; performing content comparisons; performing keyword density analysis; and performing readability analysis.

7. The method as claimed in claim 1 , further including determining if part of the first copy is invalid, and if so, removing the invalid part of the first copy to provide a verified copy of the webpage.

8. The method as claimed in claim 7 , further including indexing the verified copy.

9. The method as claimed in claim 2 , wherein the one or more properties of the first copy or the second copy at least partially include one or more of the group: partial contents of a webpage; complete contents of a webpage; links to other webpages; a check sum of the webpage; a cryptographic hash of the webpage; and a frequency of at least one keyword within the webpage.

10. A system for verifying content, comprising:

a processor;

memory in electronic communication with the processor;

the processor configured to:

receive a first copy of a webpage from a hosting server;

process the first copy to obtain first indicating data including one or more properties of the first copy;

receive second indicating data from a first client terminal, the second indicating data obtained by processing, at the first client terminal, a second copy of the webpage, the second copy received by the client terminal from the hosting server, the second indicating data including one or more properties of the second copy;

compare the first indicating data with the second indicating data;

determine that the first copy is verified when a number of non-matching properties between the first indicating data and the second indicating data does not exceed a threshold number;

determine that the first copy is not verified when the number of non-matching properties between the first indicating data and the second indicating data exceeds a threshold number;

upon determining that the first copy is not verified, receive third indicating data from a second client terminal, the third indicating data obtained by processing, at the second client terminal, a third copy of the webpage, the third copy received by the second client terminal from the hosting server, the third indicating data including one or more properties of the third copy of the webpage;

determining whether the first copy of the webpage received by the indexing system is verified or not verified based on a comparison between the first indicating data and the third indicating data; and

upon determining that the first copy is verified, indexing content of the webpage.

11. The system as claimed in claim 10 , wherein the content is webpage content.

12. The system as claimed in claim 10 , also including a database to store indexed contents of the first copy if verified.

13. The system as claimed in claim 10 , wherein the second indicating data is obtained by the at least one processing system receiving the second indicating data from the client terminal that processes the second copy.

14. A computer program product for verifying webpage content, the computer program product comprising a non-transitory storage device having instructions thereon, the instructions comprising:

code programmed to receive, at an indexing system, a first copy of a webpage from a hosting server;

code programmed to process the first copy, at the indexing system, to obtain first indicating data including one or more properties of the first copy;

code programmed to receive second indicating data from a first client terminal, the second indicating data obtained by processing, at the first client terminal, a second copy of the webpage, the second copy received by the first client terminal from the hosting server, the second indicating data including one or more properties of the second copy;

code programmed to compare, at the indexing system, the first indicating data with the second indicating data;

code programmed to determine that the first copy is verified when a number of non-matching properties between the first indicating data and the second indicating data does not exceed a threshold number;

code programmed to determine that the first copy is not verified when the number of non-matching properties between the first indicating data and the second indicating data exceeds a threshold number;

upon determining that the first copy is not verified, code programmed to receive third indicating data from a second client terminal, the third indicating data obtained by processing, at the third client terminal, a third copy of the webpage, the third copy received at the third client terminal from the hosting server, the third indicating data including one or more properties of the third copy of the webpage

code programmed to determine whether the first copy of the webpage received by the indexing system is verified or not verified based on a comparison between the first indicating data and the third indicating data; and

upon determining that the first copy is verified, code programmed to index content of the webpage.