Method and system for processing a uniform resource locator

1. A method, comprising:

generating a plurality of lattice nodes in a storage system based on a Uniform Resource Locator (URL), wherein each of the plurality of lattice nodes includes a host component corresponding to a portion of the URL and a path component corresponding to a portion of the URL;

identifying one of the plurality of lattice nodes as a search node;

searching a pattern table database using the search node to find a matching pattern table entry, the pattern table database comprising a plurality of pattern table entries corresponding to known lattice nodes associated with previously encountered URLs; and

processing the URL based on whether the matching pattern table entry was found in the pattern table database.

2. The method of claim 1 , wherein each pattern table entry comprises a pattern hash code computed based on a corresponding known lattice node and wherein searching the pattern table database comprises generating a search hash code based on the search node and wherein the pattern hash code of the matching pattern table entry matches the search hash code.

3. The method of claim 1 , comprising searching a guard table using the search node to find a corresponding guard table entry, wherein the guard table comprises a plurality of guard table entries corresponding to most general known lattice nodes based on previously encountered URLs.

4. The method of claim 1 , comprising, upon finding the matching pattern table entry, obtaining a guard state from a corresponding entry in a guard state table and searching the pattern table database using a different one of the plurality of lattice nodes based, at least in part, on the guard state.

5. The method of claim 4 , wherein each entry in the guard state table includes an indication of whether the corresponding pattern table entry corresponds to a pattern of interest and an indication of whether other lattice nodes obtained by recursively extending the search node may be patterns of interest.

6. The method of claim 1 , further comprising:

identifying in the plurality of lattice nodes, a set of child nodes of the search node and repeating the steps of searching the pattern table database, processing the URL, and identifying additional child nodes based on some or all of the child nodes being treated as further search nodes.

7. The method of claim 6 , wherein:

identifying a set of child nodes comprises adding the identified child nodes to an agenda of pending lattice nodes; and

identifying one of the plurality of lattice nodes as a search node comprises:

identifying a most-general lattice node as an initial search node;

identifying a pending lattice node in the agenda of pending lattice nodes as a subsequent search node; and

removing the identified pending lattice node from the agenda.

8. The method of claim 1 , wherein processing the URL comprises obtaining data from a corresponding entry in a data table if the matching pattern table entry was found in the pattern table database.

9. The method of claim 8 , wherein the data comprises a category indication and wherein processing the URL comprises associating the category indication with the URL.

10. The method of claim 1 , wherein processing the URL comprises:

if the matching pattern table entry was found in the pattern table database, associating the URL with a specified category; and

if the matching pattern table entry was not found in the pattern table database, not associating the URL with the specified category.

11. A computer system, comprising:

a processor that is configured to execute machine-readable instructions; and

a memory device that stores instruction modules that are executable by the processor, the instruction modules comprising:

a URL receiver configured to receive a URL;

a URL lattice generator configured to generate a plurality of lattice nodes, wherein each of the plurality of lattice nodes includes a host component corresponding to a portion of the URL and a path component corresponding to a portion of the URL;

a search module configured to identify a search node corresponding to one of the plurality of lattice nodes and search a pattern table database using the search node to find a matching pattern table entry, the pattern table database comprising a plurality of pattern table entries corresponding to known lattice nodes associated with previously encountered URLs; and

a URL processor configured to process the URL based on whether a matching pattern table entry was found in the pattern table database.

12. The computer system of claim 11 , wherein the URL lattice generator is configured to generate a first table comprising a plurality of first hash codes corresponding to components of the URL of a first component type and a second table comprising a plurality of second hash codes corresponding to components of the URL of a second component, wherein the search module is configured to identify the search node by combining one of the plurality of first hash codes from the first table with one of the plurality of second hash codes from the second table.

13. The computer system of claim 11 , wherein the URL processor is configured to obtain a guard state corresponding to the search node if the matching pattern table entry was found in the pattern table database, wherein the guard state includes a first indicator that indicates whether data corresponding to the search node may be found in a data table and a second indicator that indicates whether additional matching pattern table entries may be found in the pattern table database for sub nodes of the search node.

14. The computer system of claim 13 , comprising an agenda configured to determine which of the plurality of lattice nodes are used as search nodes:

wherein if the guard state indicates that a pattern of interest may exist for a child node of the search node, child nodes of the search node are added to the agenda; and

if the guard state indicates that the pattern of interest does not exist for any child node of the search node, the child nodes of the search node are marked as walked.

15. The computer system of claim 11 , comprising a data table generated based, at least in part, on the output of a classifier generated by processing previously encountered URLs, and wherein the URL processor is configured to receive data from the data table if the matching pattern table entry is found in the pattern table database.

16. The computer system of claim 15 , wherein the URL processor is configured to:

generate a data field hash code corresponding to a combination of the search node and a field name included in a query field of the URL;

use the data field hash code to search the data table to find a matching hash code; and

if the matching hash code is found in the data table, identify the field of the URL as a search term field.

17. A non-transitory, computer readable medium, comprising code configured to direct a processor to:

generate a plurality of lattice nodes, wherein each of the plurality of lattice nodes includes a host component corresponding to a portion of a Uniform Resource Locator (URL) and a path component corresponding to a portion of the URL;

identify one of the plurality of lattice nodes as a search node;

search a pattern table database using the search node to find a matching pattern table entry, the pattern table database comprising a plurality of pattern table entries corresponding to known lattice nodes associated with previously encountered URLs; and

process the URL based on whether the matching pattern table entry was found in the pattern table database.

18. The non-transitory, computer readable medium of claim 17 , comprising code configured to direct the processor to obtain a guard state corresponding to the search node if the matching pattern table entry was found in the pattern table database, wherein the guard state includes a first indicator that indicates whether data corresponding to the search node may be found in a data table and a second indicator that indicates whether additional matching pattern table entries may be found in the pattern table database for sub nodes of the search node.

19. The non-transitory, computer readable medium of claim 17 , comprising code configured to direct the processor to receive data from a data table if the matching pattern table entry is found in the pattern table database, wherein the data table is generated based, at least in part, on the output of a classifier generated by processing previously encountered URLs.

20. The non-transitory, computer readable medium of claim 17 , comprising code configured to direct the processor to:

generate a data field hash code corresponding to a combination of the search node and a field name included in a query field of the URL;

use the data field hash code to search the data table to find a matching hash code in the data table; and

if the matching hash code is found in the data table, identify the field of the URL as a search term field.