Surrogate hashing
Surrogate hashing is described, including a repository configured to store a standardized address and data associated with the standardized address, and a processor configured to parse an address associated with a host to generate a standardized address in a format used to compare the standardized address to one or more addresses stored in a collection, to determine if the standardized address is listed in the collection, and to process the standardized address.
1. A system, comprising:
a memory configured to store data associated with an address; and
a logic module configured:
to parse an address associated with a host,
to generate a standardized address from the address,
to determine if the standardized address is identified in a collection of URLs,
to process the standardized address to retrieve a file,
to identify a standardized portion of data contents associated with the file by identifying a data set to be selected consistently from the file, wherein the data set is identified based on a size and a location associated with the file, and to add the standardized address to the collection if the standardized address is not identified in the collection,
to run a first hashing against the standardized portion of data contents to generate a first hash value, and to run a second hashing algorithm against the standardized portion of data contents to generate a second hash value,
to determine whether the first hash value and the second hash value are equal to predetermined values associated with a first stored hash value and a second stored hash value, respectively, the first stored hash value and the second stored hash value being associated with a second portion of data contents associated with the second file, the second portion of data contents associated with the second file being similar to the first portion of data contents associated with the first file; and
to identify a location of the second file when the first hash value and the second hash value are equal to the predetermined values associated with the second portion of data contents associated with the second file.
2. The system of claim 1 , wherein the collection comprises one or more local URLs.
3. The system of claim 1 , wherein the collection comprises one or more foreign URLS.
4. The system of claim 1 , wherein the logic module is configured to determine if the standardized address is a local URL.
5. The system of claim 1 , wherein the logic module is configured to determine if the standardized address is a foreign URL.
6. The system of claim 1 , wherein the logic module is configured to identify one or more parts of the address.
7. The system of claim 6 , wherein the one or more parts of the address is a header.
8. The system of claim 6 , wherein the one or more parts of the address is data associated with a host and a file located at the address.
9. A system, comprising:
a repository configured to store a standardized address and data associated with the standardized address; and
a processor configured:
to parse an address associated with a host to generate a standardized address in a format used to compare the standardized address to one or more addresses stored in a collection of URLs,
to determine if the standardized address is listed in the collection, and
to process the standardized address to retrieve a file,
to identify a standardized portion of data contents associated with the file by identifying a data set to be selected consistently from the file, wherein the data set is identified based on a size and a location associated with the file, and to add the standardized address to the collection if the standardized address is not identified in the collection,
to run a first hashing against the standardized portion of data contents to generate a first hash value, and to run a second hashing algorithm against the standardized portion of data contents to generate a second hash value,
to determine whether the first hash value and the second hash value are equal to predetermined values associated with a first stored hash value and a second stored hash value, respectively, the first stored hash value and the second stored hash value being associated with a second portion of data contents associated with the second file, the second portion of data contents associated with the second file being similar to the first portion of data contents associated with the first file; and
to identify a location of the second file when the first hash value and the second hash value are equal to the predetermined values associated with the second portion of data contents associated with the second file.
10. The system of claim 9 , wherein the processor is configured to determine if another address is found at a location indicated by the standardized address.
11. The system of claim 9 , wherein the processor is configured to download a file associated with the address.