System, method, and computer program product for processing a prefix tree file utilizing a selected agent
View Patent ↗A system, method, and computer program product are provided for processing a prefix tree file utilizing a selected agent. In use, a file including a prefix tree is identified. Additionally, an agent is selected from a plurality of agents to process the file. Further, the file is processed utilizing the agent.
1. A method, comprising:
identifying a file including a prefix tree, wherein the prefix tree includes an ordered hierarchical data structure which stores arrays of data with a common prefix;
selecting an agent from a plurality of agents to process the file; and
processing the file utilizing this agent;
wherein identifying the file includes selecting the file from a queue of files, each file including a different prefix tree.
2. The method of claim 1 , wherein the prefix tree stores at least a portion of each of a plurality of uniform resource locators.
3. The method of claim 2 , wherein each of the uniform resource locators include the common prefix.
4. The method of claim 1 , wherein the file is stored on a client computer.
5. The method of claim 4 , wherein the plurality of agents are stored on the client computer.
6. The method of claim 1 , wherein each of the plurality of agents includes a command line executable.
7. The method of claim 1 , wherein each of the plurality of agents perform different functions.
8. The method of claim 7 , wherein each of the plurality of agents perform at least one of a crawling function, a merging function, a separating function, and a reorganization function.
9. The method of claim 1 , wherein a predefined number of the agents are capable of executing in parallel.
10. The method of claim 1 , wherein the file is selected based on a time of a last access to the file.
11. The method of claim 1 , wherein the agent is selected based on a determination of whether the agent is in an idle state.
12. The method of claim 1 , wherein the processing includes downloading a web page identified using a uniform resource locator stored in the prefix tree.
13. The method of claim 12 , wherein the processing includes analyzing the web page for gathering information associated with the uniform resource locator.
14. The method of claim 1 , wherein the processing includes categorizing a target pointed to by uniform resource locator stored in the prefix tree.
15. The method of claim 1 , wherein a plurality of different files, each file including a different prefix tree, are distributed across a plurality of devices.
16. The method of claim 15 , wherein the different files include different types of prefix trees.
17. The method of claim 1 , wherein the agent is selected based on a type of the processing to be performed on the file.
18. The method of claim 1 , wherein processing the file includes sorting the data according to common prefixes, separating the data based on the sorting, and storing the sorted data in sorted seed prefix trees, chosen based on the common prefixes.
19. A computer program product embodied on a tangible computer readable medium, comprising:
computer code for identifying a file including a prefix tree, wherein the prefix tree includes an ordered hierarchical data structure which stores arrays of data with a common prefix;
computer code for selecting an agent from a plurality of agents to process the file; and
computer code for processing the file utilizing this agent;
wherein the computer program product is operable such that identifying the file includes selecting the file from a queue of files, each file including a different prefix tree.
20. A system, comprising:
a processor for identifying a file including a prefix tree, wherein the prefix tree includes an ordered hierarchical data structure which stores arrays of data with a common prefix, the processor further adapted to select an agent from a plurality of agents to process the file and process the file utilizing this agent;
wherein the system is operable such that identifying the file includes selecting the file from a queue of files, each file including a different prefix tree.
21. The system of claim 20 , further comprising memory coupled to the processor via a bus.
22. A method, comprising:
identifying a file including a prefix tree wherein the prefix tree includes an ordered hierarchical data structure which stores arrays of data with a common prefix;
selecting an agent from a plurality of agents to process the file; and
processing the file utilizing this agent;
wherein the agent is selected based on a type of the prefix tree included in the file.
23. A computer program product embodied on a tangible computer readable medium, comprising:
computer code for identifying a file including a prefix tree, wherein the prefix tree includes an ordered hierarchical data structure which stores arrays of data with a common prefix;
computer code for selecting an agent from a plurality of agents to process the file; and
computer code for processing the file utilizing this agent;
wherein the computer program product is operable such that the agent is selected based on a type of the prefix tree included in the file.
24. A system, comprising:
a processor for identifying a file including a prefix tree, wherein the prefix tree includes an ordered hierarchical data structure which stores arrays of data with a common prefix, the processor further adapted to select an agent from a plurality of agents to process the file and process the file utilizing this agent;
wherein the system is operable such that the agent is selected based on a type of the prefix tree included in the file.