Incremental Batch Method for Transforming Event-driven Metrics and Measures within a Map/Reduce Data Center
A method for a plurality of processors configured to perform steps in a map/reduce network operation adds incremental batch transformation of sequential measures recorded by time periods and uploaded asynchronously from their capture on mobile devices. The method creates and tracks measure states for each measure. The current recurrence of a measure is a transformation of selected past recurrences and measures. Measure state is propagated according to rules. The ID for a current measure is derived from the IDs of its cache measures and the IDs of its trigger measures. A batch incremental enrichment transforms one or more measures from one or more recurrences into at least one output measure that may be transformed again by the same or another batch incremental enrichment. The apparatus determines if the value needs to be overwritten with a newer value by the type of transformation, the recurrence id and the state.
1 . An incremental batch process at a processor which does not completely reprocess the last 30 days of data in each daily batch run comprising:
retaining intermediate measures;
fully processing data collected in the period since last incremental batch process;
reprocessing intermediate measures when used to build potential new measure;
reprocessing intermediate measures when used to build a measure that may be changed;
replacing an obsolete measure;
providing a unique ID for each generated measure;
and determining when a generated measure cannot have changed.
2 . A method for incrementally enriching measures at a processor, comprising:
receiving from non-transitory data storage a previously enriched measure;
receiving a measure produced in a current recurrence( fresh measures);
determining the earliest period with fresh measures;
reading fresh measures from storage;
controlling a fresh flag on output records; and
writing out only output measures which are fresh.
3 . The method of claim 2 further comprising:
setting a flag to produce a non-fresh measure when the output is consumed by another enrichment process;
setting a recurrence attribute;
setting a new attribute to identify rows to delete; and
creating a measure-to-delete object containing at least one id.
4 . A method for a plurality of processors configured to perform steps in a map/reduce network operation to enable incremental batch transformation of sequential measures recorded by time periods and uploaded asynchronously from their capture on mobile devices, the method comprises:
creating and tracking a measure state for each measure;
creating and tracking a measure ID for each measure;
selecting and transforming at least one past recurrence of a measure into a current recurrence of the measure;
propagating a measure state from a past recurrence to a current recurrence; and
deriving a measure ID in a current recurrence from a cache measure and a trigger measure.