Joining and dimensional annotation in a streaming pipeline
Disclosed are embodiments for providing batch performance using a stream processor. In one embodiment, a method is disclosed comprising receiving, at a stream processor, an event, the stream processor including a plurality of processing stages; generating, by the stream processor, an augmented event based on the event, the augmented event including at least one additional field not appearing in the event, the additional field generated by an operation selected from the group consisting of a join or dimensional annotation operation; and emitting, by the stream processor, the augmented event to downstream consumer.
1. A method comprising:
receiving, at a stream processor, an event, the stream processor including a plurality of processing stages;
querying, by the stream processor, a join table to identify a secondary event associated with the event;
emitting, by the stream processor, the event to downstream consumer if the secondary event is not present in the join table; and
emitting, by the stream processor, an augmented event using the event and the secondary event when the secondary event is present in the join table.
2. The method of claim 1 , further comprising querying the join table to determine if the event is in the join table prior to querying the join table to identify the secondary event and writing the event to the join table if the event is not in the join table.
3. The method of claim 2 , wherein querying the join table to determine if the event is in the join table comprises extracting a join key from the event and querying the join table using the join key.
4. The method of claim 2 , wherein querying the join table to identify the secondary event and querying the join table to determine if the event is in the join table comprise a single query.
5. The method of claim 1 , further comprising labeling the secondary event if the secondary event is present in the join table.
6. The method of claim 5 , wherein labeling the secondary event comprises setting a join status header field of the secondary event in the join table.
7. The method of claim 1 , further comprising augmenting the secondary event if the secondary event is present in the join table to form the augmented event, the augmented event including one or more fields from the event.
8. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining steps of:
receiving, at a stream processor, an event, the stream processor including a plurality of processing stages;
querying, by the stream processor, a join table to identify a secondary event associated with the event;
emitting, by the stream processor, the event to downstream consumer if the secondary event is not present in the join table; and
emitting, by the stream processor, an augmented event using the event and the secondary event when the secondary event is present in the join table.
9. The non-transitory computer-readable storage medium of claim 8 , further comprising querying the join table to determine if the event is in the join table prior to querying the join table to identify the secondary event and writing the event to the join table if the event is not in the join table.
10. The non-transitory computer-readable storage medium of claim 9 , wherein querying the join table to determine if the event is in the join table comprises extracting a join key from the event and querying the join table using the join key.
11. The non-transitory computer-readable storage medium of claim 9 , wherein querying the join table to identify the secondary event and querying the join table to determine if the event is in the join table comprise a single query.
12. The non-transitory computer-readable storage medium of claim 8 , further comprising labeling the secondary event if the secondary event is present in the join table.
13. The non-transitory computer-readable storage medium of claim 12 , wherein labeling the secondary event comprises setting a join status header field of the secondary event in the join table.
14. The non-transitory computer-readable storage medium of claim 8 , further comprising augmenting the secondary event if the secondary event is present in the join table to form the augmented event, the augmented event including one or more fields from the event.
15. A device comprising:
a stream processor; and
a storage medium for tangibly storing thereon program logic for execution by the stream processor, the program logic comprising instructions for:
receiving an event, the stream processor including a plurality of processing stages;
querying a join table to identify a secondary event associated with the event;
emitting the event to downstream consumer if the secondary event is not present in the join table; and
emitting an augmented event using the event and the secondary event when the secondary event is present in the join table.
16. The device of claim 15 , the instructions further comprising querying the join table to determine if the event is in the join table prior to querying the join table to identify the secondary event and writing the event to the join table if the event is not in the join table.
17. The device of claim 16 , wherein querying the join table to determine if the event is in the join table comprises extracting a join key from the event and querying the join table using the join key.
18. The device of claim 16 , wherein querying the join table to identify the secondary event and querying the join table to determine if the event is in the join table comprise a single query.
19. The device of claim 15 , the instructions further comprising labeling the secondary event if the secondary event is present in the join table.
20. The device of claim 15 , further comprising augmenting the secondary event if the secondary event is present in the join table to form the augmented event, the augmented event including one or more fields from the event.