Bulk data processing system
The system allows real time collection and processing of massive data from many network elements. It has an elastic architecture that scales horizontally to support different network sizes. The system in a uniform data format for downstream consumption. It employs a pub/sub data distribution mechanism that supports multiple concurrent downstream subscribers efficiently in real-time.
1. A method comprising:
obtaining data from a network device, wherein the network device processes the data from a wireless network;
determining a type of the data from the network device;
based on the type of data, transforming the data into a standardized format;
partitioning the data into chunks, wherein the chunks are based on a cell identifier;
obtaining the data in the standardized format by a messaging layer, wherein the messaging layer comprises messaging layer devices that partition the data in the standardized format into a first topic; and
based on the first topic, publishing the partitioned data in the standardized format to a correlation layer, wherein the correlation layer comprises correlation layer devices that further processes the partitioned data in the standardized format to add information that places the data into a second topic.
2. The method of claim 1 , further comprising:
responsive to publishing the partitioned data in the standardized format in the first topic to the correlation layer, obtaining the partitioned data in the standardized format in the second topic; and
publishing the partitioned data in the standardized format to a subscriber of the second topic.
3. The method of claim 1 , wherein the network device is a Mobility Management Entity.
4. The method of claim 1 , wherein the network device is an eNodeB.
5. The method of claim 1 , wherein the network device is a mobile device.
6. The method of claim 1 , wherein the data from the network device is streamed.
7. A computer readable storage medium storing computer executable instructions that when executed by a computing device cause said computing device to effectuate operations comprising:
obtaining data from a network device, wherein the network device processes the data from a wireless network;
determining a type of the data from the network device;
based on the type of data, transforming the data into a standardized format;
partitioning the data into chunks, wherein the chunks are based on a type of network device;
obtaining the data in the standardized format by a messaging layer, wherein the messaging layer comprises messaging layer devices that partition the data in the standardized format into a first topic; and
based on the first topic, publishing the partitioned data in the standardized format to a correlation layer, wherein the correlation layer comprises correlation layer devices that further processes the partitioned data in the standardized format to add information that places the data into a second topic.
8. The computer readable storage medium of claim 7 , the operations further comprising:
responsive to publishing the partitioned data in the standardized format in the first topic to the correlation layer, obtaining the partitioned data in the standardized format in the second topic; and
publishing the partitioned data in the standardized format to a subscriber of the second topic.
9. The computer readable storage medium of claim 7 , wherein the network device is a Mobility Management Entity.
10. The computer readable storage medium of claim 7 , wherein the network device is an eNodeB.
11. The computer readable storage medium of claim 7 , wherein the network device is a mobile device.
12. The computer readable storage medium of claim 7 , wherein the data from the network device is streamed.
13. A system comprising:
one or more processors; and
memory coupled with the one or more processors, the memory storing executable instructions that when executed by the one or more processors cause the one or more processors to effectuate operations comprising:
obtaining data from a network device, wherein the network device processes the data from a wireless network;
determining a type of the data from the network device;
based on the type of data, transforming the data into a standardized format;
partitioning the data into chunks;
obtaining the data in the standardized format by a messaging layer, wherein the messaging layer comprises messaging layer devices that partition the data in the standardized format into a first topic; and
based on the first topic, publishing the partitioned data in the standardized format to a correlation layer, wherein the correlation layer comprises correlation layer devices that further processes the partitioned data in the standardized format to add information that places the data into a second topic.
14. The system of claim 13 , wherein the network device is a Mobility Management Entity.
15. The system of claim 13 , wherein the network device is an eNodeB.
16. The system of claim 13 , wherein the network device is a mobile device.
17. The system of claim 13 , wherein the data from the network device is streamed.
18. The system of claim 13 , the operations further comprising:
responsive to publishing the partitioned data in the standardized format in the first topic to the correlation layer, obtaining the partitioned data in the standardized format in the second topic; and
publishing the partitioned data in the standardized format to a subscriber of the second topic.
19. The system of claim 13 , the operations further comprising publishing the partitioned data in the standardized format to a subscriber of the first topic.