Pluggable join framework for stream processing
Disclosed are embodiments for providing batch performance using a stream processor. In one embodiment, a method is disclosed comprising receiving an event, such as a streaming event, from a client. The method determines that the event comprises a primary event and, if so, writes the primary event to a cache and returning the primary event to the client. The method later receives a second event from the client, the second event associated with the first event, annotates the second event based on the primary event, and returns the annotated second event to the client.
1. A method comprising:
receiving a primary event, the primary event including a join key;
identifying a secondary event associated with the primary event using the join key;
annotating the secondary event with fields from the primary event;
marking the secondary event as late-joined;
storing the primary event in a backing store;
marking the primary event and the secondary event as out-of-order; and
returning the primary event and the secondary event to a client.
2. The method of claim 1 , wherein identifying the secondary event comprises querying a backing store using the join key to identify the secondary event.
3. The method of claim 2 wherein querying the backing store comprises:
issuing a fetch command to a random-access memory (RAM) cache using the join key;
issuing, by the RAM cache, a fetch command to the backing store when the RAM cache does not include the secondary event; and
returning, by the RAM cache, the secondary event when the RAM cache includes the secondary event.
4. The method of claim 1 , further comprising marking the primary event as late-joined.
5. The method of claim 1 , wherein storing the primary event in the backing store comprises:
storing the primary event in a local cache; and
storing the primary event in the backing store upon confirmation that the primary event was stored in the local cache.
6. The method of claim 5 , wherein storing the primary event in the local cache comprises storing the primary event in a dirty cache.
7. The method of claim 1 , wherein annotating the secondary event comprises:
deserializing the secondary event to obtain a deserialized secondary event;
deserializing the primary event to obtain a deserialized primary event; and
joining the deserialized primary event and deserialized secondary event.
8. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining steps of:
receiving a primary event, the primary event including a join key;
identifying a secondary event associated with the primary event using the join key;
annotating the secondary event with fields from the primary event;
marking the secondary event as late-joined;
storing the primary event in a backing store;
marking the primary event and the secondary event as out-of-order; and
returning the primary event and the secondary event to a client.
9. The non-transitory computer-readable storage medium of claim 8 , wherein identifying the secondary event comprises querying a backing store using the join key to identify the secondary event.
10. The non-transitory computer-readable storage medium of claim 9 , wherein querying the backing store comprises:
issuing a fetch command to a random-access memory (RAM) cache using the join key;
issuing, by the RAM cache, a fetch command to the backing store when the RAM cache does not include the secondary event; and
returning, by the RAM cache, the secondary event when the RAM cache includes the secondary event.
11. The non-transitory computer-readable storage medium of claim 8 , the steps further comprising marking the primary event as late-joined.
12. The non-transitory computer-readable storage medium of claim 8 , wherein storing the primary event in the backing store comprises:
storing the primary event in a local cache; and
storing the primary event in the backing store upon confirmation that the primary event was stored in the local cache.
13. The non-transitory computer-readable storage medium of claim 12 , wherein storing the primary event in the local cache comprises storing the primary event in a dirty cache.
14. The non-transitory computer-readable storage medium of claim 8 , wherein annotating the secondary event comprises:
deserializing the secondary event to obtain a deserialized secondary event;
deserializing the primary event to obtain a deserialized primary event; and
joining the deserialized primary event and deserialized secondary event.
15. A device comprising:
a processor; and
a storage medium for tangibly storing thereon logic for execution by the processor, the logic comprising instructions for:
receiving a primary event, the primary event including a join key;
identifying a secondary event associated with the primary event using the join key;
annotating the secondary event with fields from the primary event;
marking the secondary event as late-joined;
storing the primary event in a backing store;
marking the primary event and the secondary event as out-of-order; and
returning the primary event and the secondary event to a client.
16. The device of claim 15 , wherein identifying the secondary event comprises querying a backing store using the join key to identify the secondary event.
17. The device of claim 15 , wherein querying the backing store comprises:
issuing a fetch command to a random-access memory (RAM) cache using the join key;
issuing, by the RAM cache, a fetch command to the backing store when the RAM cache does not include the secondary event; and
returning, by the RAM cache, the secondary event when the RAM cache includes the secondary event.
18. The device of claim 15 , the instructions further comprising marking the primary event as late-joined.
19. The device of claim 15 , wherein storing the primary event in the backing store comprises:
storing the primary event in a local cache; and
storing the primary event in the backing store upon confirmation that the primary event was stored in the local cache.
20. The device of claim 19 , wherein storing the primary event in the local cache comprises storing the primary event in a dirty cache.