Granted Patent B2

US 8,560,521 · App. 13/150,947 · Granted Oct 15, 2013

Abstract

Claims (31)

1. A method, comprising:

identifying a first file including a prefix tree, wherein the prefix tree includes an ordered hierarchical data structure which stores arrays of data with a common prefix, wherein the prefix tree stores at least a portion of uniform resource locators, each of the uniform resource locators having a common prefix;

selecting an agent to process the first file; and

processing the first file utilizing the agent, wherein the agent is selected based on a type of the prefix tree included in the first file, and wherein the type of the prefix tree is indicated by a file extension of the first file in which the prefix tree is included;

wherein selecting the agent includes identifying that the prefix tree is a crawl trie based upon the file extension and selecting an agent configured to perform a crawling function based upon identifying that the prefix tree is a crawl trie; and

wherein processing the first file utilizing the agent includes downloading a second file identified using a uniform resource locator stored in the prefix tree and analyzing the second file to identify unwanted content in the second file.

2. The method of claim 1 , wherein the file is stored on a client computer.

3. The method of claim 2 , wherein the agent is stored on the client computer.

4. The method of claim 1 , wherein the agent includes a command line executable.

5. The method of claim 1 , wherein the agent is selected from a pool of agents that perform different functions.

6. The method of claim 5 , wherein each of the agents in the pool of agents performs at least one of a crawling function, a merging function, a separating function, and a reorganization function.

7. The method of claim 1 , wherein a predefined number of the agents are capable of executing operations in parallel.

8. The method of claim 1 , wherein the first file is further selected based on a time of a last access to the file.

9. The method of claim 1 , wherein the agent is selected based on a determination of whether the agent is in an idle state.

10. The method of claim 1 , wherein the processing includes analyzing the second file for gathering information associated with the uniform resource locator.

11. The method of claim 1 , wherein the processing includes categorizing a target pointed to by a uniform resource locator stored in the prefix tree.

12. The method of claim 1 , wherein a plurality of different files having a different prefix tree are distributed across a plurality of devices.

13. The method of claim 12 , wherein the different files include different types of prefix trees.

14. The method of claim 1 , wherein analyzing the second file to identify unwanted content in the second file includes analyzing the second file to identify malware in the second file.

15. A method, comprising:

identifying a first file including a prefix tree, wherein the prefix tree includes an ordered hierarchical data structure which stores arrays of data with a common prefix, wherein the prefix tree stores at least a portion of uniform resource locators, each of the uniform resource locators having a common prefix;

selecting an agent to process the first file based on a type of the prefix tree included in the file, and wherein the type of the prefix tree is indicated by a file extension of the file in which the prefix tree is included, and wherein selecting the agent includes identifying that the prefix tree is a crawl trie based upon the file extension and selecting an agent configured to perform a crawling function based upon identifying that the prefix tree is a crawl trie; and

processing the first file utilizing the agent, wherein identifying the file includes selecting the file from a queue of files, and wherein processing the file utilizing the agent includes downloading a second file identified using a uniform resource locator stored in the prefix tree and analyzing the second file for gathering information associated with the uniform resource locator.

16. A non-transitory computer readable storage medium for storing instructions for performing operations comprising:

identifying a first file including a prefix tree, wherein the prefix tree includes an ordered hierarchical data structure which stores arrays of data with a common prefix, wherein the prefix tree stores at least a portion of uniform resource locators, each of the uniform resource locators having a common prefix;

selecting an agent to process the first file based on a type of the prefix tree included in the file, and wherein the type of the prefix tree is indicated by a file extension of the file in which the prefix tree is included, and wherein selecting the agent includes identifying that the prefix tree is a crawl trie based upon the file extension and selecting an agent configured to perform a crawling function based upon identifying that the prefix tree is a crawl trie; and

processing the first file utilizing the agent, wherein processing the file utilizing the agent includes downloading a second file identified using a uniform resource locator stored in the prefix tree and analyzing the second file for gathering information associated with the uniform resource locator.

17. The non-transitory computer readable storage medium of claim 16 , wherein analyzing the second file includes analyzing the second file to identify unwanted content in the second file.

18. The non-transitory computer readable storage medium of claim 16 , wherein the agent is further selected based on a determination of whether the agent is in an idle state.

19. The non-transitory computer readable storage medium of claim 16 , wherein the agent is selected from a pool of agents that perform different functions.

20. The non-transitory computer readable storage medium of claim 16 , wherein the first file is further selected based on a time of a last access to the file.

Assignments (8)