Processing Data Feeds
Exemplary embodiments allow performance of stream computations on real-time data streams using one or more map operations and/or one or more update operations. A map operation is a stream computation in which stream events in one or more real-time data streams are processed in a real-time manner to generate zero, one or more new stream events. An update operation is a stream computation in which stream events in one or more real-time data streams are processed in a real-time manner to create or update one or more static “slate” data structures that are stored in a durable manner.
1 . A computer-implemented method comprising:
receiving, at a worker process, a first stream event in a real-time data stream;
processing, at the worker process, the first stream using a map operation to generate output data;
transforming the output data, at the worker process, to generate a second stream event associated with at least one slate that records a set of related stream events; and
transmitting the second stream event in an intermediate data stream.
2 . The computer-implemented method of claim 1 , wherein the map operation is a stream computation in which stream events are processed to generate the output data.
3 . The computer-implemented method of claim 2 , wherein the stream computation determines at least one of a popular topic in the first stream, post time information, and a user's current interest.
4 . The computer-implemented method of claim 2 , wherein the stream computation includes ranking computations to determine a user's influence on other users.
5 . The computer-implemented method of claim 2 , wherein the stream computation includes analyzing data published on a web site.
6 . The computer-implemented method of claim 2 , wherein the stream computation includes grouping web page view events into visits and aggregating statistics based on visits.
7 . The computer-implemented method of claim 2 , wherein the stream computation includes determining links on a web page that maximize a click-through rate.
8 . The computer-implemented method of claim 1 , wherein the map operation is subscribed to receive a plurality of stream events in the real-time data stream.
9 . The computer-implemented method of claim 1 , further comprising: determining a computational load at a plurality of worker nodes on a network; and scheduling the worker process on a particular worker node with the lowest computation load.
10 . The computer-implemented method of claim 1 , wherein the first stream event is a unit of data transmitted in the real-time data stream, and wherein the second stream event is a unit of data transmitted in the intermediate data stream in a real-time manner.
11 . The computer-implemented method of claim 1 , wherein the at least one slate records the set of related stream events in a persistent storage device.
12 . A computer-implemented method comprising:
receiving, at a worker process, a first stream event in a real-time data stream;
processing, at the worker process, the first stream using an update operation to generate updated data;
transforming the updated data, at the worker process, to generate a second stream event associated with a set of related stream events; and
transmitting the second stream event in an intermediate data stream.
13 . The computer-implemented method of claim 12 , wherein the set of related stream event is associated with a slate that records the set of related stream events in a persistent storage device.
14 . The computer-implemented method of claim 12 , wherein the update operation is a stream computation in which stream events are processed to generate the output data.
15 . The computer-implemented method of claim 14 , wherein the stream computation determines at least one of a popular topic in the first stream, post time information, and a user's current interest.
16 . The computer-implemented method of claim 14 , wherein the stream computation includes ranking computations to determine a user's influence on other users.
17 . The computer-implemented method of claim 14 , wherein the stream computation includes analyzing data published on a web site.
18 . The computer-implemented method of claim 14 , wherein the stream computation includes grouping web page view events into visits and aggregating statistics based on visits.
19 . The computer-implemented method of claim 14 , wherein the stream computation includes determining links on a web page that maximize a click-through rate.
20 . An apparatus comprising:
a memory; and
one or more processors coupled to the memory, the one or more processors configured to:
receive a first stream event in a real-time data stream;
process the first stream using a map operation to generate output data;
transforming the output data to generate a second stream event associated with at least one slate that records a set of related stream events; and
transmitting the second stream event in an intermediate data stream.