Scaleable processing of network accounting data
View Patent ↗A data collection system includes a plurality of node host computers. Each node host computer has a node manager, at least one processing node and a data manager. The data manager delivers network records between processing nodes. The processing nodes include an input queue and an output queue. In some embodiments at least one of the nodes has a plurality of output queues. The processing nodes are arranged in configurable chains of processing nodes. The chains are disposed across one or more of the plurality of host computers to process network records. The at least one processing node having the plurality of output queues can couple records from the plurality of output queues to input queues of corresponding processing nodes, based on a selected distribution method and/or selected content of records. The system can include an administrative client that displays a graphical user interface and an administrative server communicating with the node host computers and the administrative client. The administrative server stores files of executable code that can be dynamically downloaded from the server to configure the client and change the graphical user interface dynamically while the graphical user interface is running. A node deletion process is also described.
1. A data processing node comprising:
a node manager that administers nodes and provides a run time environment for administered nodes, which include;
a node object to provide specific functional processing, the node object existing as an object within the node manager; and
a data manager that provides inter-node data communications for remote nodes or local nodes.
2. The data-processing node of claim 1 wherein the node object processes network data.
3. The data processing node of claim 1 wherein the node object is an equipment interface node that processes network data into network records.
4. The data processing node of claim 1 wherein the node object is an enhancement node that modifies a network record.
5. The data processing node of claim 1 wherein the node object is a correlation node that correlates two network records to produce an aggregated network record.
6. The data processing node of claim 1 wherein the node object is an aggregation node that aggregates multiple network records to produce an aggregated network record.
7. The data processing node of claim 1 wherein the node object is an output interface node that processes network records into data that is sent to a data consuming application.
8. The data processing node of claim 1 wherein the data manager comprises:
a configuration file used to configure the node object; and
a rules file that determines how the node object processes data.
9. The data processing node of claim 1 wherein the data manager comprises:
an input queue and an output queue for transferring network records.
10. The data processing node of claim 1 wherein the data manager comprises:
an input queue and an output queue for transferring network records; and
a persistent store, to store network records until the records are sent to a succeeding processing node.
11. The data processing node of claim 1 wherein the data manager moves data between the output queue or the input queue of the node object and to an input queue or an output queue of a succeeding node.
12. The data processing node of claim 1 wherein the data manager moves data between the output queue of the node object and to an input queue of a succeeding node.
13. The data processing node of claim 1 wherein the data manager is a local data manager that moves data from the node object to a succeeding node on a common host machine.
14. The data processing node of claim 1 wherein the data manager moves network records and the data manager is a remote data manager that moves records from the node object to a succeeding node that is on a different host machine.
15. The data processing node of claim 11 wherein the data are network accounting records.
16. The data processing node of claim 11 wherein the data are network accounting records are persisted to storage as network accounting record files.
17. A computer program product residing on a computer readable medium for producing a data collection and processing node on a computer device comprising instructions for causing a computer to:
instantiate a data processing node that comprises:
a node manager that administers nodes and provides a run time environment for administered nodes, which include;
a node object to provide specific functional processing; and
a data manager that provides inter-node data communications for remote nodes or local nodes.
18. The computer program product of claim 17 wherein the node object processes network data.
19. The computer program product of claim 17 wherein the instantiate node object is an equipment interface node that processes network data into network records.
20. The computer program product of claim 17 wherein the instantiated node object is an enhancement node that modifies the network record.
21. The computer program product of claim 1 wherein the instantiated node object is a correlation node that correlates two network records to produce an aggregated network record.
22. The computer program product of claim 17 wherein the instantiated node object is an aggregation node that aggregates multiple network records to produce an aggregated network record.
23. The computer program product of claim 17 wherein the instantiated node object is an output interface node that processes network records into data that is sent to a data consuming application.
24. The computer program product of claim 17 further comprising instructions to cause the computer to:
set up in a memory a configuration file to configure the node object and a rules file that determines how the node object processes data.
25. The computer program product of claim 17 further comprising instructions to cause the computer to:
set up in a memory a configuration file to configure the node object, a rules file that determines how the node object processes data and a persistent store to store network records until the records are sent to a succeeding processing node.
26. The computer program product of claim 17 further comprising instructions to cause the computer to cause the data manager to move data between an output queue or an input queue of the node object to an input queue or an output queue of a succeeding node.
27. The computer program product of claim 17 further comprising instructions to cause the computer to cause the data manager to move data between an output queue of the node object to an input queue a succeeding node.
28. A data collection system, comprising:
a plurality of interconnected node host computers;
a node manager resident in each of the plurality of interconnected node host computers, said node manager containing at least one processing node object and a data manager, the data manager delivering network records between a first processing node and second processing node.
29. The system of claim 28 wherein the node manager contains first and second processing node objects and within a specified one of the plurality of node host computers the node manager is a local node manager and delivers network records between the first and the second processing nodes.
30. The system of claim 28 wherein within a specified one of the plurality of node host computers the node manager is a remote data manager that delivers network accounting records between a first node object and second node object with one of the node objects being on a different host computer than the remote data manager.
31. The system of claim 28 wherein a user interacting with a graphical user interface client defines at least one chain of processing node objects for transporting at least one network accounting record through the data collection system.
32. The system of claim 28 wherein a user interacting with graphical user interface client defines multiple chains of processing nodes and wherein the multiple processing node chains operate in parallel to allow simultaneous processing of multiple network records.
33. The system of claim 28 wherein the at least one processing node object is an equipment interface nodes for collecting data from outside the data collection system and translating the data into network records.
34. The system of claim 28 wherein the at least one processing node object is an enhancement node for adding attributes to network records.
35. The system of claim 28 wherein the at least one processing node object is an aggregation processor nodes for aggregating a series of network records into a single network record.
36. The system of claim 28 wherein the at least one processing node object is an output interface node for translating network records into an external data format and delivering the translated network record to a destination outside the data collection system.
37. A data collection system comprising:
a plurality of interconnected host computers arranged to receive network data from a network;
at least some of said plurality of interconnected node host computers comprising a plurality of processing node objects to process the network data, said plurality of processing node objects arranged in at least one logical chain of processing node objects where at least some of the processing node objects on the chain perform different processing tasks on the network data.
38. The data collection system of claim 37 wherein the node objects are arranged to provide a network accounting system.
39. The data collection system of claim 37 wherein said plurality of processing node objects arranged in a plurality of logical chains of processing node objects wherein at least some of the processing node objects of one of the chains reside on different host computers.
40. The data collection system of claim 37 wherein the node objects comprise:
an input queue;
an output queue; and
wherein records are moved between the output queue of a source node to an input queue a destination node.