Automatic schema upgrade in a big data pipeline

1. A method comprising:

receiving an event, the event including a plurality of fields;

determining a set of needed fields used by a downstream processing stage of a streaming processor from the plurality of fields based on an upgrade schema migration modification;

extracting the set of needed fields from the plurality of fields;

generating a plurality of serialized fields by serializing the plurality of fields;

generating a new event, the new event including the set of needed fields and a hidden field, the value of the hidden field comprising the serialized fields;

transmitting the new event to at least one processing stage of a stream processor;

generating a reserialized event by reserializing a processed event generated by the stream processor, the processed event generated by the stream processor based on the new event; and

outputting the reserialized event to a downstream consumer.

2. The method of claim 1 , the reserializing the processed event comprising:

deserializing the hidden field to generate a deserialized event;

performing an update operation selected from the group consisting of:

updating one or more fields of the deserialized event based on the needed fields, and

adding one or more fields to the deserialized event based on the needed fields.

3. The method of claim 1 , the performing an update operation comprising selecting the update operation based on an output schema.

4. The method of claim 1 , the generating the new event further comprising inserting a value representing an emitted schema into the new event, the emitted schema comprising a schema of the received event.

5. The method of claim 4 , the reserializing the processed event comprising generating a decoder-encoder pair based on the emitted schema and an output schema associated with the emitted schema.

6. The method of claim 1 , the event comprising an exploded event, the event including a serialized copy of a combined event including the exploded event, the generating the reserialized event further comprising deleting at least one other exploded event in the combined event during the reserializing.

7. The method of claim 1 , the event comprising an exploded event, the exploded event generated by:

detecting that a combined event requires explosion; deserializing the combined event; and

generating a set of exploded events including the event, each exploded event being serialized and indexed based on the combined event.

8. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining the steps of:

receiving an event, the event including a plurality of fields;

determining a set of needed fields used by a downstream processing stage of a streaming processor from the plurality of fields based on an upgrade schema migration modification;

extracting the set of needed fields from the plurality of fields;

generating a plurality of serialized fields by serializing the plurality of fields;

generating a new event, the new event including the set of needed fields and a hidden field,

the value of the hidden field comprising the serialized fields;

transmitting the new event to at least one processing stage of a stream processor;

generating a reserialized event by reserializing a processed event generated by the stream processor, the processed event generated by the stream processor based on the new event; and

outputting the reserialized event to a downstream consumer.

9. The computer-readable storage medium of claim 8 , the reserializing the processed event comprising:

deserializing the hidden field to generate a deserialized event;

performing an update operation selected from the group consisting of:

updating one or more fields of the deserialized event based on the needed fields, and

adding one or more fields to the deserialized event based on the needed fields.

10. The computer-readable storage medium of claim 8 , the performing an update operation comprising selecting the update operation based on an output schema.

11. The computer-readable storage medium of claim 8 , the generating the new event further comprising inserting a value representing an emitted schema into the new event, the emitted schema comprising a schema of the received event.

12. The computer-readable storage medium of claim 11 , the reserializing the processed event comprising generating a decoder-encoder pair based on the emitted schema and an output schema associated with the emitted schema.

13. The computer-readable storage medium of claim 8 , the event comprising an exploded event, the event including a serialized copy of a combined event including the exploded event, the generating the reserialized event further comprising deleting at least one other exploded event in the combined event during the reserializing.

14. The computer-readable storage medium of claim 8 , the event comprising an exploded event, the exploded event generated by:

detecting that a combined event requires explosion;

deserializing the combined event; and

generating a set of exploded events including the event, each exploded event being serialized and indexed based on the combined event.

15. An apparatus comprising:

a processor; and

a storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic causing the processor to perform the operations of:

receiving an event, the event including a plurality of fields;

determining a set of needed fields used by a downstream processing stage of a streaming processor from the plurality of fields based on an upgrade schema migration modification;

extracting the set of needed fields from the plurality of fields;

generating a plurality of serialized fields by serializing the plurality of fields;

generating a new event, the new event including the set of needed fields and a hidden field, the value of the hidden field comprising the serialized fields;

transmitting the new event to at least one processing stage of a stream processor;

generating a reserialized event by reserializing a processed event generated by the stream processor, the processed event generated by the stream processor based on the new event; and

outputting the reserialized event to a downstream consumer.

16. The apparatus of claim 15 , the reserializing the processed event comprising:

deserializing the hidden field to generate a deserialized event;

performing an update operation selected from the group consisting of:

updating one or more fields of the deserialized event based on the needed fields, and

adding one or more fields to the deserialized event based on the needed fields.

17. The apparatus of claim 16 , the generating the new event further comprising inserting a value representing an emitted schema into the new event, the emitted schema comprising a schema of the received event.

18. The apparatus of claim 17 , the reserializing the processed event comprising generating a decoder-encoder pair based on the emitted schema and an output schema associated with the emitted schema.

19. The apparatus of claim 16 , the event comprising an exploded event, the event including a serialized copy of a combined event including the exploded event, the generating the reserialized event further comprising deleting at least one other exploded event in the combined event during the reserializing.

20. The apparatus of claim 16 , the event comprising an exploded event, the exploded event generated by:

detecting that a combined event requires explosion; deserializing the combined event; and

generating a set of exploded events including the event, each exploded event being serialized and indexed based on the combined event.