Scalable data enrichment for cloud streaming analytics
A method is described of pre-processing real-time data to be processed by one or more real-time analytics services. The method includes: receiving, at a stateless ingress node, data from one or more client devices; transforming the data in real-time at the stateless ingress node; determining in real-time, at the stateless ingress node, a real-time analytics service for the transformed data; and forwarding in real-time the transformed data to the determined real-time analytics service for stateful processing.
1. A method of pre-processing real-time data to be processed by one or more real-time analytics services, the method comprising:
receiving, at a stateless ingress node, data from one or more client devices, the data provided from a load balancer in front of stateless ingress nodes to balance incoming data over the stateless ingress nodes, wherein the stateless ingress node comprises one or more listeners that receive the data from the load balancer;
transforming the data in real-time at the stateless ingress node, wherein the stateless ingress node performs stateless pre-processing to output transformed data, and wherein the stateless pre-processing performed by the stateless ingress node is separate from stateful real-time analytics performed by the one or more real-time analytics services;
determining in real-time, at the stateless ingress node, a real-time analytics service for the transformed data; and
forwarding in real-time the transformed data to the determined real-time analytics service for performance of the stateful real-time analytics.
2. The method of claim 1 , wherein the data is received in the form of one or more incoming data or event streams.
3. The method of claim 1 , wherein the stateless ingress node is a virtual machine.
4. The method of claim 1 , wherein the client devices are data collecting devices and the data is telemetry data.
5. The method of claim 1 , wherein the stateless ingress node and the one or more client devices communicate via a request-response protocol.
6. The method of claim 1 , wherein the stateless ingress node has a set of data endpoints and each data endpoint supports a single protocol for communication.
7. The method of claim 1 , wherein the determining the real-time analytics service is performed, at the stateless ingress node, using a routing table stored in a metadata storage.
8. The method of claim 1 , wherein the transformation comprises performing at least one of: security operations, deserialization, verification, location enrichment, data transformation, schema changes, debatching and decompression.
9. The method of claim 1 , wherein forwarding the transformed data to the determined real-time analytics service comprises serializing the transformed data.
10. The method of claim 1 , further comprising spooling, replaying and archiving the received data.
11. The method of claim 1 , wherein the client devices are distributed systems.
12. The method of claim 1 , wherein the stateless ingress node further comprises one or more dispatchers, an event pipeline engine and a transport component.
13. The method of claim 1 , wherein a number of the stateless ingress nodes that are currently deployed is changed at runtime by a user or depending on a load.
14. The method of claim 1 , where the stateless ingress nodes are identical and independent.
15. The method of claim 1 , wherein the stateful real-time analytics are performed by a number of compute nodes.
16. The method of claim 15 , wherein the number of the compute nodes that are currently deployed is changed at runtime by a user.
17. The method of claim 1 , wherein the data is only stored in a main memory and pre-processed from the main memory during the stateless pre-processing.
18. One or more computer storage media for performing a method of pre-processing real-time data to be processed by one or more real-time analytics services comprising:
computer-executable instructions to receive, at a stateless ingress node, data from one or more data collecting devices, the data provided from a load balancer in front of stateless ingress nodes to balance incoming data over the stateless ingress nodes, wherein the stateless ingress node comprises one or more listeners that receive the data from the load balancer;
computer-executable instructions to transform the data in real-time at the stateless ingress node by performing stateless pre-processing at the stateless ingress node, and wherein the stateless pre-processing performed at the stateless ingress node is separate from stateful real-time analytics performed by the one or more real-time analytics services;
computer-executable instructions to determine in real-time, at the stateless ingress node, a real-time analytics service for the transformed data; and
computer-executable instructions to forward in real-time the transformed data to the determined real-time analytics service for performance of the stateful real-time analytics.
19. A stateless ingress node of a front-end service, wherein the stateless ingress node is arranged to perform a method of pre-processing real-time data to be processed by one or more real-time analytics services, the pre-processing comprising:
receiving data from one or more data collecting devices, the data provided from a load Balancer in front of stateless ingress nodes to balance incoming data over the stateless ingress nodes, wherein the stateless ingress node comprises one or more listeners that receive the data from the load balancer;
transforming the data in real-time;
determining in real-time a real-time analytics service for the transformed data by using a routing table; and
forwarding in real-time the transformed data to the determined real-time analytics service for stateful processing.