Semantic reflection storage and automatic reconciliation of hierarchical messages

1. A method executed at least in part in a computing device for semantic reflection storage and automatic reconciliation of hierarchical messages, the method comprising:

automatically assessing structural semantics of an incoming message;

shredding the message into its structural components;

transforming one or more structural components based on at least one of content and a structure of the message;

automatically adapting over time as incoming messages from a known source change;

generating a table for each segment of the shredded message at a view materializer based on data from a first generation data store, a second generation data store, and a correlation data store, wherein shredding is performed automatically placing the one or more structural components into a first generation data store in insert-only form, and a user is enabled to manage exceptions to a default behavior through a configuration; and

publishing the generated table.

2. The method of claim 1 , further comprising:

storing the structural components at the first generation data store, wherein the message is shredded by a first generation parser component.

3. The method of claim 2 , wherein transforming the structural components comprises:

deriving one or more components from the structural components; and

generating correlation data mapping context keys of the structural components to entities.

4. The method of claim 3 , further comprising:

storing the derived components and the structural components at the second generation data store, wherein the message derivation and correlation are performed by a second generation parser component.

5. The method of claim 4 , further comprising:

generating a view for each segment of the shredded message at the view materializer based on data from the first generation data store, the second generation data store, and the correlation data store;

storing the at least one view and table at an aggregate data store.

6. The method of claim 5 , further comprising:

binding aggregated results to entities using the correlation data prior to publishing the entities to an entity data store.

7. The method of claim 1 , wherein the configuration specifies at least

one from a set of: elements within a first generation segment to be read for derivation, derivation results, and derivation types.

8. The method of claim 1 , further comprising:

retrieve a plurality of messages from a message queue for each feed type;

shredding the plurality of messages in parallel; and

writing resulting data tables to the first generation data store in a batch using table valued parameters.

9. A computing device for managing semantic reflection storage and automatic reconciliation of hierarchical messages, the computing device comprising:

a memory; a processor coupled to the memory, the processor executing a message processing application in conjunction with instructions stored in the memory, wherein the message processing application includes:

a first generation parser configured to automatically assess structural semantics of an incoming message and shred the message into its structural components;

a second generation parser configured to perform at least one from a set of: cleansing, normalization, and typed transformation on the structural components based on at least one of a content and a structure of the message, wherein the second generation parser is further configured to generate an incremental collapsed result that takes into account messages with a same incremental context key; and

a view materializer configured to generate a table for each segment of the shredded message based on data from a first generation data store, a second generation data store, and a correlation data store, generate feed collapsed data aggregated from feeds as a table, and store the feed collapsed data in an aggregate store.

10. The computing device of claim 9 , wherein the first generation parser is further configured to employ context keys that define entities as a concatenation of values that exist within each feed.

11. The computing device of claim 9 , wherein the second generation parser is further configured to perform at least one of: a code look-up and a composite transform.

12. The computing device of claim 9 , wherein a table structure of the table for each segment includes at least one from a set of: basic columns, shredded columns, derived columns, and context key columns.

13. The computing device of claim 9 , wherein the message processing application is further configured to:

store shredded message components in the first generation data store as segment tables; and

store a set of tables created for each segment table in the first generation data store.

14. The computing device of claim 9 , wherein the transformation is performed as an additive auto-adaptation process that adds new columns for each newly encountered segment of a shredded message.

15. A computer-readable memory device with instructions stored thereon for managing semantic reflection storage and automatic reconciliation of hierarchical messages, the instructions comprising:

performing a structural reflection process by:

identifying a highest level delimiter with each item discovered representing a segment of a received message, wherein segments are derived by shredding the received message into its structural components,

reflecting each segment for a next level of the delimiter, and

recognizing each discovered item as a field;

performing a storage provision process by adapting a structural semantic description of the shredded message to be able to provision database storage;

performing an auto-adaptation process by inspecting the received message and verifying discrepancies against an existing database storage model for structural components of the received message such that if one of a new segment, a new field, and a new component is discovered, the database storage is automatically modified to provide a storage container for a new structural component.

16. The computer-readable memory device of claim 15 , wherein the storage provision process employs a mechanism that is at least one from a set of: a combination of tables, columns, and extensible markup language (XML) columns.

17. The computer-readable memory device of claim 15 , wherein the instructions further comprise:

repeating the structural reflection process until all levels of structural semantics are discovered.