System and method for processing network data
View Patent ↗Methods and systems for providing data analytics and generating real-time and historical views of network events using a single processing pipeline, managed by a single code base, are presented. A computing device may receive a stream of data indicative of a plurality of events occurring on a network. The computing device may process the stream of data to generate intermediate data and batch data using the single processing pipeline. The intermediate data may be available to generate historical views and the batch data may include a plurality of intermediate data for a time interval. The computing device may generate a historical view of the events based on a subset of intermediate data and the batch data. Finally, the computing device may provide the historical view to a processing layer to enable the computing device to respond to requests for information about the network.
1. A computer-implemented method comprising:
identifying, using a single processing pipeline, data types and protocols of data indicative of a plurality of events of a network based on a schema defined by an input source;
generating intermediate and batch data, the intermediate data being generated from the data indicative of the plurality of events of the network and the batch data being generated from at least a subset of the intermediate data, the generation of the intermediate and batch data being done in series with one another by use of the single processing pipeline;
generating a historical view of the events of the network based on the generated intermediate and batch data; and
providing a response to requests for information about the network using the historical view.
2. The computer-implemented method of claim 1 , comprising:
presenting events occurring on the network to be presented, wherein the events comprise at least one data point corresponding to the intermediate data.
3. The computer-implemented method of claim 1 , comprising:
storing the intermediate data in a first memory of a computing device comprising a plurality of intermediate data; and
storing the historical view in a temporal database of the computing device.
4. The computer-implemented method of claim 1 , comprising:
adjusting one or more network parameters based on the intermediate data.
5. The computer-implemented method of claim 1 , comprising:
adjusting one or more network parameters based on the batch data.
6. The computer-implemented method of claim 1 , further comprising:
receiving a stream of data comprising the data indicative of the plurality of events of the network.
7. The computer-implemented method of claim 1 , further comprising:
building, using machine learning, a model of the network, wherein the model is used to determine one or more abnormal conditions.
8. A computing device, comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the computing device to:
identify, using a single processing pipeline, data types and protocols of data indicative of a plurality of events of a network based on a schema defined by an input source;
generate intermediate and batch data, the intermediate data being generated from the data indicative of the plurality of events of the network and the batch data being generated from at least a subset of the intermediate data, the generation of the intermediate and batch data being done in series with one another by use of the single processing pipeline;
generate a historical view of the events of the network based on the generated intermediate and batch data; and
provide a response to requests for information about the network using the historical view.
9. The computing device of claim 8 , wherein the instructions, when executed by the one or more processors, cause the computing device to:
present events occurring on the network, wherein the events comprise at least one data point corresponding to the intermediate data.
10. The computing device of claim 8 , wherein the instructions, when executed by the one or more processors, cause the computing device to:
store the intermediate data in a first memory of the computing device comprising a plurality of intermediate data; and
store the historical view in a temporal database of the computing device.
11. The computing device of claim 8 , wherein the instructions, when executed by the one or more processors, cause the computing device to:
adjust one or more network parameters based on the intermediate data.
12. The computing device of claim 8 , wherein the instructions, when executed by the one or more processors, cause the computing device to:
adjust one or more network parameters based on the batch data.
13. The computing device of claim 8 , wherein the instructions, when executed by the one or more processors, cause the computing device to:
receive a stream of data comprising the data indicative of the plurality of events of the network.
14. The computing device of claim 8 , wherein the instructions, when executed by the one or more processors, cause the computing device to:
build, using machine learning, a model of the network, wherein the model is used to determine one or more abnormal conditions.
15. A non-transitory computer-readable medium comprising instructions that, when executed by a computing device, cause the computing device to:
identify, using a single processing pipeline, data types and protocols of data indicative of a plurality of events of a network based on a schema defined by an input source;
generate intermediate data and batch data, the intermediate data being generated from the data indicative of the plurality of events of the network and the batch data being generated from at least a subset of the intermediate data, the generation of the intermediate and batch data being done in series with one another by use of the single processing pipeline;
generate a historical view of the events of the network based on the generated intermediate and batch data; and
provide a response to requests for information about the network using the historical view.
16. The non-transitory computer-readable medium of claim 15 , wherein the instructions, when executed by the computing device, cause the computing device to:
present events occurring on the network, wherein the events comprise at least one data point corresponding to the intermediate data.
17. The non-transitory computer-readable medium of claim 15 , wherein the instructions, when executed by the computing device, cause the computing device to:
store the intermediate data in a first memory of the computing device comprising a plurality of intermediate data; and
store the historical view in a temporal database of the computing device.
18. The non-transitory computer-readable medium of claim 15 , wherein the instructions, when executed by the computing device, cause the computing device to:
adjust one or more network parameters based on the intermediate data.
19. The non-transitory computer-readable medium of claim 15 , wherein the instructions, when executed by the computing device, cause the computing device to:
adjust one or more network parameters based on the batch data.
20. The non-transitory computer-readable medium of claim 15 , wherein the instructions, when executed by the computing device, cause the computing device to:
build, using machine learning, a model of the network, wherein the model is used to determine one or more abnormal conditions.