System for fully integrated capture, and analysis of business information resulting in predictive decision making and simulation
A system for fully integrated collection of business impacting data, analysis of that data and generation of both analysis driven business decisions and analysis driven simulations of alternate candidate business action comprising a business data retrieval engine stored in a memory of and operating on a processor of a computing device, a business data analysis engine stored in a memory of and operating on a processor of a computing device and a business decision and business action path simulation engine stored in a memory of and operating on a processor of one of more computing devices has been developed.
1. A system for automated business decision optimization, comprising:
a network-connected computing device comprising a memory and a processor;
a high volume deep web scraper comprising first plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the first plurality of programming instructions, when operating on the processor, cause the network-connected computing device to perform deep web searches by extracting information from data stores located on the Internet that are not accessible by conventional web crawlers;
a business data retrieval engine comprising second plurality of programming instructions_stored in the memory of, and operating on the processor of, the network-connected computing device, wherein the second plurality of programming instructions, when operating on the processor, cause the network-connected computing device to:
receive a business analysis campaign configuration comprising analysis parameters for optimizing a business decision;
retrieve a plurality of business operations data from devices within a business computer network related to the business analysis campaign configuration;
direct the high volume deep web scraper to retrieve a plurality of supplemental data from deep web extraction related to the business analysis campaign configuration;
a directed computational graph module comprising a third plurality of programming instructions stored in the memory of, and operating on the processor of, the network-connected computing device, wherein the third plurality of programming instructions, when operating on the processor, cause the network-connected computing device to:
construct a directed computational graph from the business analysis campaign configuration, wherein:
the directed computational graph comprises nodes representing data transformations and edges representing messages between the nodes; and
the nodes and edges of the directed computational graph represent data processing pipelines for analyzing the business analysis campaign configuration; and
a business decision and business action path simulation engine comprising a fourth plurality of programming instructions stored in the memory of, and operating on the processor of, the network-connected computing device, wherein the fourth plurality of programming instructions, when operating on the processor, cause the network-connected computing device to:
determine a set of possible prospective actions from the business analysis campaign configuration, the business operations data, and the supplemental data;
simulate the outcome of each prospective action using the data processing pipelines of the directed computing graph as a simulation model;
determine an optimal outcome from the parametric analysis by matching the outcome of each prospective action against the analysis parameters; and
recommend the prospective action with the optimal outcome as the business decision.
2. The system of claim 1 , wherein the business information retrieval engine employs a portal for human interface device to input the current business analysis campaign configuration.
3. The system of claim 2 , wherein high volume deep web scraper receives at least some scrape control and spider configuration parameters from a customizable cloud-based interface, coordinates one or more world wide web searches (scrapes) using both general search control parameters and individual web search agent (spider) specific configuration data, receives scrape progress feedback information which may lead to issuance of further scrape control and spider configuration parameters, controls and monitors the spiders on distributed scrape servers, receives raw scrape campaign data from scrape servers, aggregates at least portions of scrape campaign data from each web site or web page traversed as per the parameters of the scrape campaign data.
4. The system of claim 3 , wherein the scrape control and spider configuration parameters are provided by a library of spider templates and individual spiders are created using the spider templates.
5. The system of claim 3 , wherein one or more of the business analysis campaign configurations is persistently stored and can be reused or used as basis for similar business analysis campaigns.
6. The system of claim 1 , wherein the business information retrieval engine employs a multidimensional time series data store comprising a fifth plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the fifth plurality of programming instructions, when operating on the processor, cause the network-connected computing device to receive a plurality of data from a plurality of sensors of heterogeneous types, some of which may have heterogeneous reporting and data payload transmission profiles, aggregates the sensor data over a predetermined amount of time, a predetermined quantity of data, or a predetermined number of events, retrieves a specific quantity of aggregated sensor data per each access connection predetermined to allow reliable receipt and inclusion of the data, transparently retrieves quantities of aggregated sensor data too large to be reliably transferred by one access connection using a further plurality access connections to allow capture of all aggregated sensor data under conditions of heavy sensor data influx and stores aggregated sensor data in a simple key-value pair with very little or no data transformation from how the aggregated sensor data is received.
7. The system of claim 1 , wherein the directed computational graph module:
retrieves streams of input from one or more of a plurality of data sources;
filters data to remove data records from each stream based on one or more of the following:
absence of all information, damage to data in the record, and presence of in-congruent information or missing information which invalidates the data record;
splits filtered data stream into two or more identical streams;
formats data within one data stream based upon a set of predetermined parameters; and
sends identical data stream for further analysis and either linear transformation or branching transformation using resources of the system.
8. A method for automated business decision optimization, comprising the steps of:
receiving a business analysis campaign configuration comprising analysis parameters for optimizing a business decision;
retrieving a plurality of business operations data from devices within a business computer network related to the business analysis campaign configuration;
directing a high volume web crawler to retrieve a plurality of supplemental data from deep web extraction related to the business analysis campaign configuration by extracting information from data stores located on the Internet that are not accessible by conventional web crawlers;
constructing a directed computational graph from the business analysis campaign configuration, wherein:
the directed computational graph comprises nodes representing data transformations and edges representing messages between the nodes; and
the nodes and edges of the directed computational graph represent data processing pipelines for analyzing the business analysis campaign configuration;
determining a set of possible prospective actions from the business analysis campaign configuration, the business operations data, and the supplemental data;
simulating the outcome of each prospective action using the data processing pipelines of the directed computing graph as a simulation model;
determining an optimal outcome from the parametric analysis by matching the outcome of each prospective action against the analysis parameters; and
recommending the prospective action with the optimal outcome as the business decision.
9. The method of claim 8 , wherein a portal for human interface device is used to input the business analysis campaign configuration.
10. The method of claim 9 , wherein the high volume deep web scraper receives at least some scrape control and spider configuration parameters from a customizable cloud-based interface, coordinates one or more world wide web searches (scrapes) using both general search control parameters and individual web search agent (spider) specific configuration data, receives scrape progress feedback information which may lead to issuance of further scrape control and spider configuration parameters, controls and monitors the spiders on distributed scrape servers, and receives raw scrape campaign data from scrape servers, aggregates at least portions of scrape campaign data from each web site or web page traversed as per the parameters of the scrape campaign data.
11. The method of claim 10 , wherein the scrape control and spider configuration parameters are provided by a library of spider templates and individual spiders are created using the spider templates.
12. The method of claim, 10 wherein one or more of the business analysis campaign configurations is persistently stored and can be reused or used as basis for similar business analysis campaigns.
13. The method of claim 9 , wherein the a multidimensional time series data is used to receive a plurality of data from a plurality of sensors of heterogeneous types, some of which may have heterogeneous reporting and data payload transmission profiles, aggregates the sensor data over a predetermined amount of time, a predetermined quantity of data or a predetermined number of events, retrieves a specific quantity of aggregated sensor data per each access connection predetermined to allow reliable receipt and inclusion of the data, transparently retrieves quantities of aggregated sensor data too large to be reliably transferred by one access connection using a further plurality access connections to allow capture of all aggregated sensor data under conditions of heavy sensor data influx and stores aggregated sensor data in a simple key-value pair with very little or no data transformation from how the aggregated sensor data is received.
14. The method of claim 8 , wherein the directed computational graph module:
retrieves streams of input from one or more of a plurality of data sources;
filters data to remove data records from the stream based on one or more of the following:
absence of all information, damage to data in the record, and presence of in-congruent information or missing information which invalidates the data record;
splits filtered data stream into two or more identical streams, formats data within one data stream based upon a set of predetermined parameters so as to prepare for meaningful storage in a data store; and
sends identical data stream for further analysis and either linear transformation or branching transformation using resources of the system.