Systems and Methods for Robust, Incremental Data Ingest of Communications Networks Topology

1 . A method for managing network elements in a telecommunications network, comprising:

processing, by a real-time processing module, stream data representing changes to one or more network elements in the telecommunications network to build a real-time view of the telecommunications network;

saving the real-time view in a view database;

processing, by a batch processing module, snapshot data representing all the network elements in the telecommunications network to build a batch view of the telecommunications network, wherein the batch processing module processes the snapshot data slower than the real-time processing module processes the stream data; and

saving the batch view in the view database.

2 . The method of claim 1 , wherein the processing the stream data comprises:

building the real-time view of the telecommunications network by applying the changes, represented by the stream data, to a most recent view in the view database.

3 . The method of claim 1 , further comprising:

selecting a view from the view database based on selection criteria; and

presenting the selected view for display to a user.

4 . The method of claim 1 , further comprising:

presenting a ranked list of views for display to a user, wherein an order of the ranked list is based on ranking criteria;

receiving a user selection of a view in the ranked list; and

presenting the selected view for display to the user.

5 . The method of claim 1 , wherein the processing the stream data comprises:

receiving the stream data;

parsing the stream data to convert the stream data into data in a common format;

storing the converted data in a first database associated with the real-time processing module; and

building the real-time view from the first database based on the converted data and a most recent view in the view database.

6 . The method of claim 1 , wherein the processing the snapshot data comprises:

receiving the snapshot data;

parsing the snapshot data to identify key-value pairs from the snapshot data;

mapping the identified key-value pairs to network entity attributes in a network model; and

storing the mapped network entity attributes in a second database associated with the batch processing module; and

building the batch view from the second database based on the mapped network entity attributes.

7 . The method of claim 6 , wherein the processing the snapshot data further comprises:

auditing data in the second database to check data consistencies.

8 . A system for managing network elements in a telecommunications network, comprising:

a memory; and

at least one processor coupled to the memory and configured to:

process, by a real-time processing module, stream data representing changes to one or more network elements in the telecommunications network to build a real-time view of the telecommunications network;

save the real-time view in a view database;

process, by a batch processing module, snapshot data representing all the network elements in the telecommunications network to build a batch view of the telecommunications network, wherein the batch processing module processes the snapshot data slower than the real-time processing module processes the stream data; and

save the batch view in the view database.

9 . The system of claim 8 , wherein the at least one processor is configured to build the real-time view by:

building the real-time view of the telecommunications network by applying the changes, represented by the stream data, to a most recent view in the view database.

10 . The system of claim 8 , wherein the at least one processor is further configured to:

select a view from the view database based on selection criteria; and

present the selected view for display to a user.

11 . The system of claim 8 , wherein the at least one processor is further configured to:

present a ranked list of views for display to a user, wherein an order of the ranked list is based on ranking criteria;

receive a user selection of a view in the ranked list; and

present the selected view for display to the user.

12 . The system of claim 8 , wherein the at least one processor is further configured to process the stream data by:

receiving the stream data;

parsing the stream data to convert the stream data into data in a common format;

storing the converted data in a first database associated with the real-time processing module; and

building the real-time view from the first database based on the converted data and a most recent view in the view database.

13 . The system of claim 8 , wherein the at least one processor is further configured to process the snapshot data by:

receiving the snapshot data;

parsing the snapshot data to identify key-value pairs from the snapshot data;

mapping the identified key-value pairs to network entity attributes in a network model; and

storing the mapped network entity attributes in a second database associated with the batch processing module; and

building the batch view from the second database based on the mapped network entity attributes.

14 . The system of claim 13 , wherein the at least one processor is further configured to process the snapshot data by:

auditing data in the second database to check data consistencies.

15 . A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations for managing network elements in a telecommunications network, comprising:

processing, by a real-time processing module, stream data representing changes to one or more network elements in the telecommunications network to build a real-time view of the telecommunications network;

saving the real-time view in a view database;

processing, by a batch processing module, snapshot data representing all the network elements in the telecommunications network to build a batch view of the telecommunications network, wherein the batch processing module processes the snapshot data slower than the real-time processing module processes the stream data; and

saving the batch view in the view database.

16 . The non-transitory computer-readable medium of claim 15 , wherein the processing the stream data comprises:

building the real-time view of the telecommunications network by applying the changes, represented by the stream data, to a most recent view in the view database.

17 . The non-transitory computer-readable medium of claim 15 , further comprising:

selecting a view from the view database based on selection criteria; and

presenting the selected view for display to a user.

18 . The non-transitory computer-readable medium of claim 15 , further comprising:

presenting a ranked list of views for display to a user, wherein an order of the ranked list is based on ranking criteria;

receiving a user selection of a view in the ranked list; and

presenting the selected view for display to the user.

19 . The non-transitory computer-readable medium of claim 15 , wherein the processing the stream data comprises:

receiving the stream data;

parsing the stream data to convert the stream data into data in a common format;

storing the converted data in a first database associated with the real-time processing module; and

building the real-time view from the first database based on the converted data and a most recent view in the view database.

20 . The non-transitory computer-readable medium of claim 15 , wherein the processing the snapshot data comprises:

receiving the snapshot data;

parsing the snapshot data to identify key-value pairs from the snapshot data;

mapping the identified key-value pairs to network entity attributes in a network model; and

storing the mapped network entity attributes in a second database associated with the batch processing module; and

building the batch view from the second database based on the mapped network entity attributes.

21 . The non-transitory computer-readable medium of claim 15 , wherein the processing the snapshot data further comprises:

auditing data in the second database to check data consistencies.