Backend data aggregation system and method
A backend data aggregation and customer experience (CX) systems and methods for processing very-large enterprise CX data. A columnar data store has a transform processor coupled thereto via a data communication network. The transform processor is configured to receive customer experience data from a plurality of data sources and to dynamically transform the received data into a columnar data schema format for storage in the columnar data store. A dynamic Object-Relational Mapper (ORM) is configured to dynamically map the transformed data to structured data based on metadata associated therewith and to store the structured data in the columnar data store.
1. A backend data aggregation system for processing very-large enterprise customer experience data comprising:
at least one backend processor configured to receive customer experience data from a plurality of data sources via the data communication network;
a database coupled to the at least one backend processor via a data communication network, the database storing the data received from the plurality of data sources in a relational format, the data received from the plurality of data sources having metadata associated therewith;
a columnar data store coupled to the at least one backend processor via the data communication network; and
at least one non-transitory memory device coupled to the backend processor via the data communication network, the memory device storing computer-executable instructions that, when executed by the backend processor, configure the backend processor to dynamically transform the data received from the plurality of data sources based on the metadata associated therewith from the relational format into a columnar data schema format suitable for storage in the columnar data store and to populate the columnar data store with the transformed data; and
wherein the memory device further stores a dynamic Object-Relational Mapper (ORM) that, when executed by the backend processor, configures the backend processor to dynamically map the transformed data in the columnar data store to structured data based on the metadata associated therewith for retrieval from the columnar data store and aggregation by the backend processor in response to a query.
2. The system of claim 1 , wherein the at least one backend processor comprises a customer experience (CX) processor coupled to the columnar data store via the data communication network, and wherein the memory devices further store computer-executable instructions that, when executed by the CX processor, configure the CX processor to receive a request for a query from a user via a web application and parse the query into a structured query language.
3. The system of claim 2 , wherein the at least one backend processor comprises a query processor coupled to the columnar data store via the data communication network, and wherein the memory devices further store computer-executable instructions that, when executed by the query processor, configure the query processor to execute the parsed query against the columnar data store to retrieve structured data responsive to the query.
4. The system of claim 3 , further comprising a display coupled to the query processor via the data communication network, the display configured to display the retrieved structured data in an aggregated format for the user.
5. The system of claim 2 , wherein the memory devices further store computer-executable instructions that, when executed by the CX processor, configure the CX processor to parse the query as a function of the metadata associated with the structured data stored in the columnar data store.
6. The system of claim 5 , wherein the memory devices further store computer-executable instructions that, when executed by the CX processor, configure the CX processor to translate the query into Structured Query Language (SQL) using the metadata.
7. The system of claim 1 , wherein the at least one backend processor comprises a transform processor, and wherein the memory devices further store an Extract Transform Load process that, when executed by the transform processor, configure the transform processor to dynamically transform the data into the columnar data schema format.
8. The system of claim 7 , further comprising a messaging cluster coupled to the transform processor via the data communication network, the messaging cluster configured to perform stream processing on the data received from the plurality of data sources, wherein the transform processor dynamically transforms the stream processed data into the columnar data schema format.
9. A method of processing very-large enterprise customer experience data comprising:
receiving customer experience data from a plurality of data sources;
storing the data received from the plurality of data sources in a relational format, the data received from the plurality of data sources having metadata associated therewith;
dynamically transforming, by at least one backend processor, the data received from the plurality of data sources based on the metadata associated therewith from the relational format into a columnar data schema format suitable for storage in a columnar data store, the backend processor and the columnar data store coupled to the plurality of data sources and to each other via a data communication network;
populating the columnar data store with the transformed data;
dynamically mapping, by a dynamic Object-Relational Mapper (ORM) executed by the backend processor, the transformed data in the columnar data store to structured data based on metadata associated therewith; and
retrieving and aggregating the structured data in the columnar data store by the backend processor in response to a query.
10. The method of claim 9 , further comprising receiving, by the backend processor, a request for a query from a user via a web application and parsing, by the backend processor, the query into a structured query language.
11. The method of claim 10 , further comprising executing, by the backend processor, the parsed query against the columnar data store to retrieve structured data responsive to the query.
12. The method of claim 11 , further comprising displaying the retrieved structured data in an aggregated format for the user.
13. The method of claim 10 , wherein parsing the query comprises parsing the query as a function of the metadata associated with the structured data stored in the columnar data store.
14. The method of claim 13 , further comprising translating the query into Structured Query Language (SQL) using the metadata.
15. The method of claim 9 , wherein transforming the data comprises executing an Extract Transform Load process to dynamically transform the data into the columnar data schema format.
16. The method of claim 9 , further comprising performing, by the backend processor, stream processing on the data received from the plurality of data sources, wherein transforming the data comprises dynamically transforming the stream processed data into the columnar data schema format.
17. A customer experience (CX) reporting system comprising:
one or more backend processors;
a columnar data store coupled to the one or more backend processors via a data communication network;
a database coupled to the one or more backend processors via the data communication network;
one or more memory devices coupled to the one or more backend processors via the data communication network, the one or more memory devices storing computer-executable instructions that, when executed by the one or more backend processors, configure the one or more backend processors to:
receive customer experience data from a plurality of data sources coupled to the one or more backend processors via the data communication network;
store the data received from the plurality of data sources in the database in a relational format, the data received from the plurality of data sources having metadata associated therewith;
dynamically transform, by an Extract Transform Load process executed by the one or more backend processors, the data received from the plurality of data sources based on the metadata associated therewith from the relational format into a columnar data schema format suitable for storage in the columnar data store;
populate the columnar data store with the transformed data;
dynamically map, by an Object-Relational Mapper executed by the one or more backend processors, the transformed data in the columnar data store to structured data based on metadata associated therewith; and
retrieve and aggregate the structured data in the columnar data store by the one or more backend processors in response to a query; and
a display coupled to the one or more backend processors via the data communication network, the display configured to display, in an aggregated format, the structured data retrieved from the columnar data store in response to the query.
18. The system of claim 17 , wherein the computer-executable instructions, when executed by the one or more backend processors, further configure the one or more backend processors to receive a request for the query from a user via a web application, parse the query into a structured query language, execute the parsed query against the columnar data store to retrieve structured data responsive to the query, and display the retrieved structured data in the aggregated format for the user.
19. The system of claim 18 , wherein the computer-executable instructions, when executed by the one or more backend processors, further configure the one or more backend processors to parse the query as a function of the metadata associated with the structured data stored in the columnar data store.
20. The system of claim 17 , wherein the computer-executable instructions, when executed by the one or more backend processors, further configure the one or more backend processors to perform stream processing on the data received from the plurality of data sources and to dynamically transform the stream processed data into the columnar data schema format.