Scalable event stream data processing using a messaging system
A system processes streaming data and includes at least one processor. The system may write streaming data received from a data source as messages in queues at a queuing cluster. The queuing cluster includes a coordinator node to direct the messages to non-coordinator nodes of the queuing cluster. The system may retrieve the data from the queues based on subscription of topics and store the retrieved data in a consumable repository.
1. A system for processing data to generate consumable data for downstream consumers comprising:
at least one processor configured to:
write streaming data received from a data source as messages in queues at a queuing cluster, wherein the queuing cluster includes a coordinator node to direct the messages to non-coordinator nodes of the queuing cluster;
retrieve the data from the queues based on subscription of topics;
store the retrieved data in a consumable repository;
wherein the at least one processor is further configured to:
listen for the streaming data to be available in the queues by a processing node, the streaming data being event data for one or more events at the data source;
read each event from the queues by the processing node; and
transform the events to a data model of the consumable repository.
2. The system of claim 1 , wherein the at least one processor is further configured to:
manage the processing node by a coordinator node of a processing cluster; and
retry processing of the streaming data on a different processing node until successful in response to the processing node encountering a failure.
3. The system of claim 1 , wherein the streaming data includes patient data received from a healthcare network.
4. The system of claim 2 , wherein the consumable repository is a queuing system including a coordinator node to direct the processing node to write the retrieved data to a non-coordinator node in the consumable repository.
5. The system of claim 4 , wherein the queuing cluster and consumable repository each include multiple non-coordinator nodes that provide automated replication and failover through management by the respective coordinator nodes.
6. The system of claim 1 , wherein the at least one processor is further configured to perform one or more operations from a group of:
routing the retrieved data to different long term storage; and
pushing the retrieved data to a number of different services that satisfy specific business needs.
7. The system of claim 1 , wherein the at least one processor is further configured to concurrently retrieve data from the consumable repository by a plurality of consumers, wherein the consumers maintain respective pointers to track data that each consumer has processed.
8. A computer program product for linking data objects for common entities across source systems, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by at least one processor to cause the at least one processor to:
write streaming data received from a data source as messages in queues at a queuing cluster, wherein the queuing cluster includes a coordinator node to direct the messages to non-coordinator nodes of the queuing cluster;
retrieve the data from the queues based on subscription of topics;
store the retrieved data in a consumable repository;
wherein the at least one processor is further caused to:
listen for the streaming data to be available in the queues by a processing node, the streaming data being event data for one or more events at the data source;
read each event from the queues by the processing node; and
transform the events to a data model of the consumable repository.
9. The computer program product of claim 8 , wherein the at least one processor is further caused to:
manage the processing node by a coordinator node of a processing cluster; and
retry processing of the streaming data on a different processing node until successful in response to the processing node encountering a failure.
10. The computer program product of claim 8 , wherein the streaming data includes patient data received from a healthcare network.
11. The computer program product of claim 9 , wherein the consumable repository is a queuing system including a coordinator node to direct the processing node to write the retrieved data to a non-coordinator node in the consumable repository.
12. The computer program product of claim 11 , wherein the queuing cluster and consumable repository each include multiple non-coordinator nodes that provide automated replication and failover through management by the respective coordinator nodes.
13. The computer program product of claim 8 , wherein the at least one processor is further caused to perform one or more operations from a group of:
routing the retrieved data to different long term storage; and
pushing the retrieved data to a number of different services that satisfy specific business needs.
14. The computer program product of claim 8 , wherein the at least one processor is further caused to concurrently retrieve data from the consumable repository by a plurality of consumers, wherein the consumers maintain respective pointers to track data that each consumer has processed.