Method and apparatus for collecting and storing statistics data from network elements using scalable architecture
A method and apparatus for collecting statistics data over a communications network using scalable stateless processes are disclosed. A process capable of obtaining statistics data, in one embodiment, detects a delay associated with the availability of statistics data in accordance with predefined lag duration, and subsequently, adds one or more stateless processes to a pool of stateless processes to increase capacity of collecting statistics data. Upon enabling a stateless process to collect statistics data from network element (“NE”) in response to a statistics collecting tasks, the statistics data is stored in a database.
1. A method for collecting statistics data over a communications network, comprising:
detecting a delay associated with availability of statistics data collected from a plurality of network elements (“NEs”) in accordance with a predefined time period;
determining insufficient processing capacity of stateless processes to complete collecting statistics data from all nodes within predefined time duration in response to the delay;
adding one or more additional stateless processes to a pool of stateless processes for scaling up computing power in response to the delay to increase capacity of collecting statistics data;
enabling one of the pool of stateless processes to collect statistics data from an NE in response to one of a list of statistics collecting tasks, wherein enabling one of the pool of stateless processes includes fetching the statistics data in response to a data collection manager (“DCM”) task generated by a scheduler; and
storing the statistics data in a database.
2. The method of claim 1 , further comprising,
fetching the statistics data from the database; and
generating a status report associated with the NE in accordance with the statistics data.
3. The method of claim 1 , further comprising generating a comma-separated values (“CSV”) file based on information stored in the database.
4. The method of claim 1 , wherein detecting a delay associated with availability of statistics data further includes:
identifying a predefined waiting period needed for fetching the statistics data from an NE for a predefined time interval; and
determining duration of the delay is greater than the predefined waiting period.
5. The method of claim 1 , wherein adding one or more additional stateless processes to a plurality of stateless processes includes coupling an additional computer, having one or more stateless data collection manager (“DCM”) containers capable of processing one or more DCM tasks, to a group of computers containing a plurality of DCM containers.
6. The method of claim 1 , wherein enabling one of the plurality of stateless process to collect statistics data from an NE includes obtaining the statistics data from a node over a network employing file transfer protocol (“FTP).
7. The method of claim 1 , wherein fetching the statistics data in response to a data collection manager (“DCM”) task includes generating the DCM task in response to input from orchestrator.
8. The method of claim 7 , further comprising activating the scheduler to generate information relating to time of each DCM task to be executed.
9. The method of claim 8 , further comprising enabling the orchestrator to generate information relating to tasks listed in the DCM task.
10. The method of claim 1 , wherein storing the statistics data in a database includes storing the statistics data in a customized partitioned Oracle® database.
11. A communications network, comprising:
a scheduler and orchestrator module in a network server coupled to the communications network and configured to generate a list of data collecting tasks for obtaining statistics data from various network nodes, wherein the scheduler and orchestrator module is configured to determine insufficient processing capacity of stateless processes for collecting statistics data from all nodes within predefined time duration;
a pool of stateless processes residing in one or more network devices coupled to the scheduler and orchestrate module and configured to collect information from the various network nodes in accordance with the list of data collecting tasks, wherein a stateless process is dynamically added to or removed from the pool of stateless processes for increasing or reducing capacity of collecting statistics data in accordance with computing power needed to perform data collecting within a predefined time period; and
a database coupled to the plurality of stateless processes and capable of storing the statistics data collected from one or more network nodes.
12. The network of claim 11 , further includes a scalable module coupled to the scheduler and orchestrator and operable to add additional stateless processes to the plurality of stateless processes when processing capacity of the plurality of the stateless processes is insufficient to collect statistics data from the network nodes within a predefined time interval.
13. The network of claim 12 , wherein each network node is a network elements (“NEs”) including a computer, a server, a mobile device, or a cluster of systems coupled to the communications network.
14. The network of claim 13 , wherein a list of data collecting tasks is a list of data collection manager (“DCM”).
15. The network of claim 11 , wherein the database is an Oracle® database which is partitioned in multiple levels to minimize performance interference between a write operation and a read operation of statistics data.
16. A method for collecting statistics over a communications network, comprising:
partitioning a relational database into a daily level and an interval level;
detecting a delay associated with availability of statistics data collected from a plurality of network elements (“NEs”) in accordance with a predefined expected period;
determining insufficient processing capacity of stateless processes to complete collecting statistics data from all nodes within predefined time duration in response to the delay;
adding one or more stateless processes to a pool of existing stateless processes for scaling up computing power in response to the delay to increase capacity of processing power;
collecting statistics data from NEs of the communications network in response to a list of tasks; wherein collecting statistics data includes fetching statistics data in response to a data collection manager (“DCM”) task generated by a scheduler; and
storing the statistics data in the relational database in accordance with the daily level and the interval level.
17. The method of claim 16 , further comprising,
searching and fetching the statistics data stored in the relational database; and
generating a status report in accordance with the statistics data.
18. The method of claim 16 , wherein partitioning a relational database into a daily level and an interval level further includes:
allocating a portion of Oracle® relational database in accordance with a calendar dates; and
allocating another portion of Oracle® relational database in accordance with a collection time interval.
19. The method of claim 16 , wherein detecting a delay associated with availability of statistics data in accordance with a predefined expected duration further includes:
identifying duration of the predefined expected period which is time needed to fetch the statistics data; and
determining duration of the delay is greater than the duration of the predefined expected period.
20. The method of claim 16 , wherein adding one or more stateless processes to a pool of existing stateless processes includes coupling an additional computer, having one or more stateless data collection manager (“DCM”) containers capable of processing one or more DCM tasks, to a group of computers containing a plurality of DCM containers.