SYSTEM AND METHOD FOR OPERATING A BIG-DATA PLATFORM
A system and method for operating a big-data platform that includes at a data analysis platform, receiving discrete client data; storing the client data in a network accessible distributed storage system that includes: storing the client data in a real-time storage system; and merging the client data into a columnar-based distributed archive storage system; receiving a data query request through a query interface; and selectively interfacing with the client data from the real-time storage system and archive storage system according to the query.
1 . A method for operating a big-data platform comprising:
at a data analysis platform, receiving discrete client data;
storing the client data in a network accessible distributed storage system that includes:
storing the client data in a real-time storage system;
merging the client data into a columnar-based distributed archive storage system;
receiving a data query request through a query interface; and
selectively interfacing with the client data from the real-time storage system and archive storage system according to the query.
2 . The method of claim 1 , wherein selectively interfacing with the client data includes at a data-intensive processing cluster, processing the data query according to a data mapping and reduction processes in querying data from the real-time storage system and archive storage system.
3 . The method of claim 2 , wherein the data-intensive processing cluster is a Hadoop cluster and the data mapping and reduction process is a MapReduce implementation.
4 . The method of claim 1 , wherein discrete client data is received and stored with dynamic schema.
5 . The method of claim 4 , wherein the data query request includes a schema definition and wherein selectively interfacing with the client data includes applying the schema definition to the dynamic schema.
6 . The method of claim 1 , further comprising at a client data agent collecting client data and transmitting the client data to the data analysis platform.
7 . The method of claim 6 , wherein the client data agent is integrated into an event channel from which client data is collected.
8 . The method of claim 7 , wherein the event channel is selected from the list consisting of syslog, a relational database, cloud data, and sensor data.
9 . The method of claim 6 , further comprising at the client data agent serializing data into a binary serialization data-interchange that is transmitted to the data analysis platform.
10 . The method of claim 6 , wherein collecting client data is collected through a client agent data-input plugin.
11 . The method of claim 1 , wherein the columnar-based distributed archive storage system stores client data in time series order, and wherein selectively interfacing with client data includes querying data from distributed storage system.
12 . The method of claim 11 , wherein querying client data from distributed storage system includes cooperatively querying the real-time storage system and the archive storage system for a cohesive query result.
13 . The method of claim 1 , wherein receiving a data query includes converting relational database styled query to data-intensive cluster query process.
14 . The method of claim 1 , wherein the data query request is received through a infographics interface and further comprising returning an infographic from the selectively interfaced client data.
15 . The method of claim 1 , wherein receiving a data query includes receiving the data query through a business intelligence tool driver and further comprising returning data analytics results to the business intelligence tool driver.
16 . The method of claim 1 , wherein client data is associated with a user account through unique identifier.
17 . The method of claim 16 , [scalable querying and isolated data storage], wherein client data merged into the archive data storage is isolated according to the user account associated with the client data and a query processing cluster interfaces with the distributed storage system, and the query processing cluster is shared between by a plurality of user accounts.
18 . The method of claim 1 , further comprising at a client data agent collecting client data and transmitting the client data to the data analysis platform; wherein the columnar-based distributed archive storage system stores client data in time series order with a dynamic schema, and wherein selectively interfacing with client data includes cooperatively querying data from the real-time storage system and the archive storage system for a cohesive query result.
19 . The method of claim 18 , wherein distributed storage system includes over one petabyte of data.