Method And System For Pre-Processing Data Received From Data Sources To Deduce Meaningful Information
Disclosed is perceptual system for facilitating a stream of data required for data analytics is disclosed. The perceptual system comprises a data ingestion governance module, a data input ports module, a data ingestion module, and an opportunity input packager module. The data ingestion governance module receives an input instruction, from a data analytics system, associated to data required for data analytics. The data ingestion governance module further determines a data source, amongst a plurality of data sources, generating a stream of data. The data input ports module establishes a network connection with the data source in order to extract the stream of data. The data ingestion module transforms the stream of data into a stream of transformed data based on the input instruction. The opportunity input packager module generates a stream of input packets upon including metadata in the stream of transformed data and thereby transmitting the stream of input packets to the data analytics system configured to perform the data analytics on the data.
1 . A method for facilitating a stream of data required for data analytics, the method comprising:
receiving, by a processor, an input instruction, from a data analytics system, associated to data required for data analytics;
determining, by a processor, a data source, amongst a plurality of data sources, generating a stream of data, wherein the data source is determined based on a predefined mapping of a type of the data analytics, to be performed, with one or more data sources of the plurality of data sources;
establishing, by the processor, a network connection with the data source in order to extract the stream of data;
transforming, by the processor, the stream of data into a stream of transformed data based on the input instruction, wherein the stream of data is transformed upon performing at least one data transformation technique; and
generating, by the processor, a stream of input packets upon including metadata in the stream of transformed data and thereby transmitting the stream of input packets to the data analytics system configured to perform the data analytics on the data in a cognitive decision-making process.
2 . The method of claim 1 , wherein the input instruction comprises a type of data, one or more data fields to be included for the data analytics, and a frequency of receiving the data.
3 . The method of claim 1 , wherein the determination of the data source includes a network path of the data source, procedure to establish connection to the data source, and procedure to extract data from the data source.
4 . The method of claim 1 , wherein the at least one data transformation technique comprises fixing of data transmission errors, correcting semantic errors in raw data, including missing data, and changing data format.
5 . The method of claim 1 , wherein the metadata comprises pre-defined business policies and business rules, models developed from historical data, and hyper-parameters of models learned over time.
6 . A perceptual system for facilitating a stream of data required for data analytics, the perceptual system comprising:
a processor; and
a memory coupled to the processor, wherein the processor is capable of executing a plurality of modules stored in the memory, and wherein the plurality of modules comprising:
a data ingestion governance module for
receiving an input instruction, from a data analytics system, associated to data required for data analytics, and
determining a data source, amongst a plurality of data sources, generating a stream of data, wherein the data source is determined based on a predefined mapping of a type of the data analytics, to be performed, with one or more data sources of the plurality of data sources;
a data input ports module for establishing a network connection with the data source in order to extract the stream of data;
a data ingestion module for transforming the stream of data into a stream of transformed data based on the input instruction, wherein the stream of data is transformed upon performing at least one data transformation technique; and
an opportunity input packager module for generating a stream of input packets upon including metadata in the stream of transformed data and thereby transmitting the stream of input packets to the data analytics system configured to perform the data analytics on the data in a cognitive decision-making process.
7 . The perceptual system of claim 6 , wherein the input instruction comprises a type of data, one or more data fields to be included for the data analytics, and a frequency of receiving the data.
8 . The perceptual system of claim 6 , wherein the data ingestion governance module further determines a network path of the data source, procedure to establish connection to the data source, and procedure to extract data from the data source.
9 . The perceptual system of claim 6 , wherein the data ingestion module transforms the stream of data by at least one of fixing of data transmission errors, correcting semantic errors in raw data, including missing data, and changing data format.
10 . The perceptual system of claim 6 , wherein the metadata comprises pre-defined business policies and business rules, models developed from historical data, and hyper-parameters of models learned over time.
11 . A non-transitory computer readable medium embodying a program executable in a computing device for facilitating a stream of data required for data analytics, the program comprising a program code:
a program code for receiving an input instruction, from a data analytics system, associated to data required for data analytics;
a program code for determining a data source, amongst a plurality of data sources, generating a stream of data, wherein the data source is determined based on a predefined mapping of a type of the data analytics, to be performed, with one or more data sources of the plurality of data sources;
a program code for establishing a network connection with the data source in order to extract the stream of data;
a program code for transforming the stream of data into a stream of transformed data based on the input instruction, wherein the stream of data is transformed upon performing at least one data transformation technique; and
a program code for generating a stream of input packets upon including metadata in the stream of transformed data and thereby transmitting the stream of input packets to the data analytics system configured to perform the data analytics on the data in a cognitive decision-making process.