Pluggable join framework for stream processing
Disclosed are embodiments for providing batch performance using a stream processor. In one embodiment, a method is disclosed comprising receiving an event, such as a streaming event, from a client. The method determines that the event comprises a primary event and, if so, writes the primary event to a cache and returning the primary event to the client. The method later receives a second event from the client, the second event associated with the first event, annotates the second event based on the primary event, and returns the annotated second event to the client.
1. A method comprising:
receiving a first event from a client, the first event comprising a streaming event;
determining that the first event comprises a primary event;
writing the first event to a cache and returning the first event to the client;
receiving a second event from the client, the second event associated with the first event;
annotating the second event based on the first event;
returning the annotated second event to the client;
determining that the event comprises a secondary event and that a corresponding primary event was not written to the cache;
marking the secondary event as un-joined;
writing the secondary event to the cache; and
returning the marked secondary event to the client.
2. The method of claim 1 , further comprising writing the first event to a backing store after the writing the first event to the cache.
3. The method of claim 2 , further comprising:
receiving the corresponding primary event from the client;
retrieving the secondary event from the cache;
annotating the secondary event using the primary event;
marking the corresponding primary event and the secondary event as out-of-order; and
returning the corresponding primary and secondary event to the client.
4. The method of claim 2 , further comprising writing the secondary event to a delay cache and sleeping until receiving the corresponding primary event.
5. The method of claim 1 , further comprising marking the first event as in-order and returning the first event to the client before receiving the second event.
6. The method of claim 1 , the determining the first event comprises a primary event comprising querying the cache for one or more secondary events.
7. An apparatus comprising:
a processor; and
a storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic causing the processor to perform the operations of:
receiving a first event from a client, the first event comprising a streaming event;
determining that the first event comprises a primary event;
writing the first event to a cache and returning the first event to the client;
receiving a second event from the client, the second event associated with the first event;
annotating the second event based on the first event;
returning the annotated second event to the client;
determining that the event comprises a secondary event and that a corresponding primary event was not written to the cache;
marking the secondary event as un-joined;
writing the secondary event to the cache; and
returning the marked secondary event to the client.
8. The apparatus of claim 7 , the operations further comprising writing the first event to a backing store after the writing the first event to the cache.
9. The apparatus of claim 7 , the operations further comprising marking the first event as in-order and returning the first event to the client before receiving the second event.
10. The apparatus of claim 7 , the determining the first event comprises a primary event comprising querying the cache for one or more secondary events.
11. The apparatus of claim 7 , the operations further comprising:
receiving the corresponding primary event from the client;
retrieving the secondary event from the cache;
annotating the secondary event using the primary event;
marking the corresponding primary event and the secondary event as out-of-order; and
returning the corresponding primary and secondary event to the client.
12. The apparatus of claim 7 , the operations further comprising writing the secondary event to a delay cache and sleeping until receiving the corresponding primary event.
13. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining the steps of:
receiving a first event from a client, the first event comprising a streaming event;
determining that the first event comprises a primary event;
writing the first event to a cache and returning the first event to the client;
receiving a second event from the client, the second event associated with the first event;
annotating the second event based on the first event;
returning the annotated second event to the client;
determining that the event comprises a secondary event and that a corresponding primary event was not written to the cache;
marking the secondary event as un-joined;
writing the secondary event to the cache; and
returning the marked secondary event to the client.
14. The computer-readable storage medium of claim 13 , the instructions further defining the step of writing the first event to a backing store after the writing the first event to the cache.
15. The computer-readable storage medium of claim 13 , the instructions further defining the step of marking the first event as in-order and returning the first event to the client before receiving the second event.
16. The computer-readable storage medium of claim 13 , the determining the first event comprises a primary event comprising querying the cache for one or more secondary events.
17. The computer-readable storage medium of claim 13 , the instructions further defining the steps of:
receiving the corresponding primary event from the client;
retrieving the secondary event from the cache;
annotating the secondary event using the primary event;
marking the corresponding primary event and the secondary event as out-of-order; and
returning the corresponding primary and secondary event to the client.