Pluggable join framework for stream processing
Disclosed are embodiments for providing batch performance using a stream processor. In one embodiment, a method is disclosed comprising receiving an event, such as a streaming event, from a client. The method determines that the event comprises a primary event and, if so, writes the primary event to a cache and returning the primary event to the client. The method later receives a second event from the client, the second event associated with the first event, annotates the second event based on the primary event, and returns the annotated second event to the client.
1. A method comprising:
receiving a streaming event from a client;
annotating the streaming event based on a primary streaming event received prior to the streaming event and returning the annotated streaming event to the client;
determining that the streaming event comprises a secondary event and that the primary streaming event was not written to a cache;
marking the streaming event as un-joined;
writing the streaming event to the cache; and
returning the marked streaming event to the client.
2. The method of claim 1 , further comprising writing the primary streaming event to the cache and returning the primary streaming event to the client.
3. The method of claim 2 , further comprising writing the primary streaming event to a backing store after the writing the primary streaming event to the cache.
4. The method of claim 2 , further comprising marking the primary streaming event as in-order and returning the primary streaming event to the client before receiving the streaming event.
5. The method of claim 2 , the further comprising determining the primary streaming event comprises a primary event by querying the cache for one or more secondary events.
6. The method of claim 2 , further comprising:
retrieving the streaming event from the cache;
annotating the streaming event using the primary streaming event;
marking the primary streaming event and the streaming event as out-of-order; and
returning the primary streaming event and streaming event to the client.
7. The method of claim 1 , further comprising writing the streaming event to a delay cache and sleeping until receiving a corresponding primary event.
8. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining steps of:
receiving a streaming event from a client;
annotating the streaming event based on a primary streaming event received prior to the streaming event and returning the annotated streaming event to the client;
determining that the streaming event comprises a secondary event and that the primary streaming event was not written to a cache;
marking the streaming event as un-joined;
writing the streaming event to the cache; and
returning the marked streaming event to the client.
9. The non-transitory computer-readable storage medium of claim 8 , the steps further comprising writing the primary streaming event to the cache and returning the primary streaming event to the client.
10. The non-transitory computer-readable storage medium of claim 9 , further comprising writing the primary streaming event to a backing store after the writing the primary streaming event to the cache.
11. The non-transitory computer-readable storage medium of claim 9 , further comprising marking the primary streaming event as in-order and returning the primary streaming event to the client before receiving the streaming event.
12. The non-transitory computer-readable storage medium of claim 9 , the steps further comprising determining the primary streaming event comprises a primary event comprising querying the cache for one or more secondary events.
13. The non-transitory computer-readable storage medium of claim 9 , the steps further comprising:
retrieving the streaming event from the cache;
annotating the streaming event using the primary streaming event;
marking the primary streaming event and the streaming event as out-of-order; and
returning the primary streaming event and streaming event to the client.
14. The non-transitory computer-readable storage medium of claim 8 , the steps further comprising writing the streaming event to a delay cache and sleeping until receiving a corresponding primary event.
15. An apparatus comprising a processor configured to:
receive a streaming event from a client;
annotating the streaming event based on a primary streaming event received prior to the streaming event and returning the annotated streaming event to the client;
determine that the streaming event comprises a secondary event and that the primary streaming event was not written to a cache;
mark the streaming event as un-joined;
write the streaming event to the cache; and
return the marked streaming event to the client.
16. The apparatus of claim 15 , the processor further configured to write the primary streaming event to the cache and returning the primary streaming event to the client.
17. The apparatus of claim 16 , the processor further configured to write the primary streaming event to a backing store after the writing the primary streaming event to the cache.
18. The apparatus of claim 16 , the processor further configured to mark the primary streaming event as in-order and return the primary streaming event to the client before receiving the streaming event.
19. The apparatus of claim 16 , the processor further configured to determine the primary streaming event comprises a primary event by querying the cache for one or more secondary events.
20. The apparatus of claim 15 , the processor further configured to write the streaming event to a delay cache and sleeping until receiving a corresponding primary event.