Communicating URL categorization information

1. A system, comprising:

a processor configured to:

receive a URL categorization query including at least one received URL, wherein the received URL comprises a hostname portion and a path portion, and wherein the hostname portion of the received URL comprises a plurality of words separated by one or more delimiters;

use the received URL to determine a set of one or more data distribution keys, including by reversing an order of the words in the hostname portion of the received URL;

query a distributed key-value data store using the determined set of data distribution keys, wherein the distributed key-value data store comprises a plurality of master nodes, and wherein the query of the distributed key-value data store is performed at a first master node based on the hostname portion, wherein additional operations associated with the hostname portion are also performed at the first master node, and wherein operations associated with a different hostname portion of a different URL are performed at a second master node; and

return categorization information; and

a memory coupled to the processor and configured to provide the processor with instructions.

2. The system of claim 1 , wherein returning the categorization information includes determining a total number of child URLs under a node.

3. The system of claim 1 , wherein returning the categorization information includes determining a number of direct child nodes under a node.

4. The system of claim 1 , wherein returning the categorization information includes sending all child URLs.

5. The system of claim 1 , wherein returning the categorization information includes sending all direct child URLs.

6. The system of claim 1 , wherein returning the categorization information includes sending a Bloom filter bitmap.

7. The system of claim 6 , wherein the Bloom filter bitmap comprises information associated one or more direct child nodes.

8. The system of claim 7 , wherein the Bloom filter bitmap is constructed using a set of hashes computed for each direct child node.

9. The system of claim 1 , wherein returning the categorization information includes sending categorization for a single URL.

10. The system of claim 1 , wherein the URL categorization query is received from a policy enforcement appliance.

11. The system of claim 10 , wherein the policy enforcement appliance is configured to take an enforcement action with respect to the at least one URL based at least in part on the returned categorization information.

12. The system of claim 10 , wherein the policy enforcement appliance is configured to query the system when a child URL has a Bloom filter bitmap hit.

13. The system of claim 1 , wherein the set of one or more data distribution keys comprises a plurality of keys, and wherein returning categorization information includes determining which categorization information for which key in the set should be returned.

14. The system of claim 13 , wherein determining which categorization information should be returned includes determining which key corresponds to a longest URL.

15. The system of claim 13 , wherein determining which categorization information should be returned includes determining which key corresponds to a malicious categorization.

16. The system of claim 15 , wherein determining which key corresponds to a malicious categorization includes determining a severity of the malicious categorization.

17. A method, comprising:

receiving a URL categorization query including at least one received URL, wherein the received URL comprises a hostname portion and a path portion, and wherein the hostname portion of the received URL comprises a plurality of words separated by one or more delimiters;

using the received URL to determine a set of one or more data distribution keys, including by reversing an order of the words in the hostname portion of the received URL;

querying a distributed key-value data store using the determined set of data distribution keys, wherein the distributed key-value data store comprises a plurality of master nodes, and wherein the query of the distributed key-value data store is performed at a first master node based on the hostname portion, wherein additional operations associated with the hostname portion are also performed at the first master node, and wherein operations associated with a different hostname portion of a different URL are performed at a second master node; and

returning categorization information.

18. The method of claim 17 , wherein returning the categorization information includes determining a total number of child URLs under a node.

19. The method of claim 17 , wherein returning the categorization information includes determining a number of direct child nodes under a node.

20. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

receiving a URL categorization query including at least one received URL, wherein the received URL comprises a hostname portion and a path portion, and wherein the hostname portion of the received URL comprises a plurality of words separated by one or more delimiters;

using the received URL to determine a set of one or more data distribution keys, including by reversing an order of the words in the hostname portion of the received URL;

querying a distributed key-value data store using the determined set of data distribution keys, wherein the distributed key-value data store comprises a plurality of master nodes, and wherein the query of the distributed key-value data store is performed at a first master node based on the hostname portion, wherein additional operations associated with the hostname portion are also performed at the first master node, and wherein operations associated with a different hostname portion of a different URL are performed at a second master node; and

returning categorization information.

21. The method of claim 17 , wherein returning the categorization information includes sending all child URLs.

22. The method of claim 17 , wherein returning the categorization information includes sending all direct child URLs.

23. The method of claim 17 , wherein returning the categorization information includes sending a Bloom filter bitmap.

24. The method of claim 23 , wherein the Bloom filter bitmap comprises information associated one or more direct child nodes.

25. The method of claim 24 , wherein the Bloom filter bitmap is constructed using a set of hashes computed for each direct child node.

26. The method of claim 17 , wherein returning the categorization information includes sending categorization for a single URL.

27. The method of claim 17 , wherein the URL categorization query is received from a policy enforcement appliance.

28. The method of claim 27 , wherein the policy enforcement appliance is configured to take an enforcement action with respect to the at least one URL based at least in part on the returned categorization information.

29. The method of claim 27 , wherein the policy enforcement appliance is configured to query the system when a child URL has a Bloom filter bitmap hit.

30. The method of claim 17 , wherein the set of one or more data distribution keys comprises a plurality of keys, and wherein returning categorization information includes determining which categorization information for which key in the set should be returned.

31. The method of claim 30 , wherein determining which categorization information should be returned includes determining which key corresponds to a longest URL.

32. The method of claim 30 , wherein determining which categorization information should be returned includes determining which key corresponds to a malicious categorization.

33. The method of claim 32 , wherein determining which key corresponds to the malicious categorization includes determining a severity of the malicious categorization.