Bulk data processing system
The system allows real time collection and processing of massive data from many network elements. It has an elastic architecture that scales horizontally to support different network sizes. The system in a uniform data format for downstream consumption. It employs a pub/sub data distribution mechanism that supports multiple concurrent downstream subscribers efficiently in real-time.
1. A method comprising:
obtaining data from a network device, wherein the network device processes data from a wireless network;
determining a type of the data from the network device;
based on the type of the data, transforming the data into a standardized format;
partitioning the data into chunks, wherein the chunks are based on a cell identifier;
publishing the partitioned data in the standardized format in a first topic to a correlation layer;
responsive to publishing the partitioned data in the standardized format in the first topic to the correlation layer, obtaining the partitioned data in the standardized format in a second topic; and
publishing the partitioned data in the standardized format to a subscriber of the second topic.
2. The method of claim 1 , wherein the data is obtained using a collection layer that comprises a plurality of collector devices that decodes the data into the standardized format from a vendor format.
3. The method of claim 1 , further comprising obtaining the data in the standardized format by a messaging layer, wherein the messaging layer partitions the data in the standardized format into the first topic, wherein the first topic is based on an International Mobile Subscriber Identity.
4. The method of claim 1 , further comprising:
obtaining the data in the standardized format by a messaging layer, wherein the messaging layer partitions the data in the standardized format into the first topic; and
publishing the partitioned data in the standardized format to the subscriber of the first topic.
5. The method of claim 1 , wherein the correlation layer comprises correlation layer devices that further processes the partitioned data in the standardized format to add information that places the data into the second topic.
6. The method of claim 1 , wherein the network device is a Mobility Management Entity.
7. The method of claim 1 , wherein the network device is a base station.
8. The method of claim 1 , wherein the network device is a mobile device.
9. The method of claim 1 , wherein the data from the network device is streamed.
10. A system comprising:
one or more processors; and
memory coupled with the one or more processors, the memory storing executable instructions that when executed by the one or more processors cause the one or more processors to effectuate operations comprising:
obtaining data from a network device, wherein the network device processes data from a wireless network;
determining a type of the data from the network device;
based on the type of the data, transforming the data into a standardized format;
partitioning the data into chunks, wherein the chunks are based on a cell identifier;
publishing the partitioned data in the standardized format in a first topic to a correlation layer;
responsive to publishing the partitioned data in the standardized format in the first topic to the correlation layer, obtaining the partitioned data in the standardized format in a second topic; and
publishing the partitioned data in the standardized format to a subscriber of the second topic.
11. The system of claim 10 , wherein the network device is a Mobility Management Entity.
12. The system of claim 10 , wherein the network device is a base station.
13. The system of claim 10 , wherein the network device is a mobile device.
14. The system of claim 10 , wherein the data from the network device is streamed.
15. The system of claim 10 , wherein the data is obtained using a collection layer that comprises a plurality of collector devices that decodes the data into the standardized format from a vendor format.
16. The system of claim 10 , the operations further comprising obtaining the data in the standardized format by a messaging layer, wherein the messaging layer partitions the data in the standardized format into the first topic, wherein the first topic is based on an International Mobile Subscriber Identity.
17. The system of claim 10 , the operations further comprising:
obtaining the data in the standardized format by a messaging layer, wherein the messaging layer partitions the data in the standardized format into the first topic; and
publishing the partitioned data in the standardized format to the subscriber of the first topic.
18. The system of claim 10 , wherein the correlation layer comprises correlation layer devices that further processes the partitioned data in the standardized format to add information that places the data into the second topic.
19. A tangible computer readable storage medium storing computer executable instructions that when executed by a computing device cause said computing device to effectuate operations comprising:
obtaining data from a network device, wherein the network device processes data from a wireless network;
determining a type of the data from the network device;
based on the type of the data, transforming the data into a standardized format;
partitioning the data into chunks, wherein the chunks are based on a cell identifier;
publishing the partitioned data in the standardized format in a first topic to a correlation layer;
responsive to publishing the partitioned data in the standardized format in the first topic to the correlation layer, obtaining the partitioned data in the standardized format in a second topic; and
publishing the partitioned data in the standardized format to a subscriber of the second topic.
20. The tangible computer readable storage medium of claim 19 , wherein the network device is a base station.