Automated data extraction and adaptation
Systems and methods for automated data extraction and adaptation are disclosed. The system may receive a data input from an external source using various different input channels. The system may determine a data quality of the data input by comparing data fields of the data input to known metadata in the system. The system may reformat the data input based on the comparison to a format consumable by downstream applications and services. The system may apply various machine learning operations on the data input including a descriptive analytics analysis, a predictive learning analysis, and/or a prescriptive intelligence analysis.
1 . A method performed by a computer-based system, the method comprising:
in response to a computer-based system determining that a format of a data input from an external source is unknown based on a comparison of the format of the data input to a format of a data structure in a database, the method further comprises:
generating a predicted format for the data input using one or more machine learning models that have been trained on the data structure;
reformatting the data input to generate a reformatted data input based on the predicted format, wherein the reformatted data input comprises a standardized format comprising data fields of a specific data input type in a defined order;
storing the reformatted data input in the database;
training each of the one or more machine learning models with the reformatted data input;
moving one or more data fields of the data input to comply with the standardized format;
generating a machine learning analysis output;
in response to the machine learning analysis output, transmitting the machine learning analysis output to the external source associated with the input channel to cause the external source to enhance future data inputs into the input channel; and
receiving an enhanced data input from the external source, wherein the enhanced data input was generated using the machine learning analysis output.
2 . The method of claim 1 , wherein the generating the machine learning analysis output further comprises:
generating, by the computer-based system, a descriptive analytics output in response to a first data field of the data input corresponding to the format of the data structure;
assessing, by the computer-based system, the data input using a predictive learning analysis to determine a second data field of the data input that is not included in the format of the data structure; and
generating, by the computer-based system, the machine learning analysis output using a prescriptive intelligence analysis based on the descriptive analytics output and the predictive learning analysis.
3 . The method of claim 2 , further comprising:
enhancing, by the computer-based system, the input channel based on the machine learning analysis output.
4 . The method of claim 2 , further comprising:
generating, by the computer-based system, a recommendation message based on the machine learning analysis output.
5 . The method of claim 1 , wherein the input channel comprises at least one of:
an API input, a web service input, a web portal input, or a file feed input, and
wherein the data input is received using middleware.
6 . The method of claim 1 , wherein the input channel comprises a physical document input, and
wherein the data input is received using optical character recognition (OCR).
7 . The method of claim 1 , wherein the input channel comprises at least one of:
a speech input or a text input, and
wherein the data input is received using at least one of a gateway or a natural language processing (NLP) module.
8 . The method of claim 7 , wherein the input channel comprises an email input, and
the data input is received using at least one email webhook or the NLP module.
9 . The method of claim 1 , wherein the data input comprises at least one of:
transaction data or merchant data, and
wherein the reformatted data input is consumed to comply with a know your customer (KYC) regulation, a financial risk rating, an anti-money laundering law, or a financial legal requirement.
10 . A system comprising:
a processor; and
a non-transitory memory having instructions stored thereon that, in response to execution by the processor, configure the processor to:
in response to a determination that a format of a data input from an external source is unknown based on a comparison of the format of the data input to a format of a data structure in a database, the instructions further configure the processor to:
generate a predicted format for the data input using one or more machine learning models that have been trained on the data structure;
reformat the data input to generate a reformatted data input based on the predicted format, wherein the reformatted data input comprises a standardized format comprising data fields of a specific data input type in a defined order;
store the reformatted data input in the database;
train each of the one or more machine learning models with the reformatted data input;
move one or more data fields of the data input to comply with the standardized format;
generate a machine learning analysis output;
in response to the machine learning analysis output, transmit the machine learning analysis output to the external source associated with the input channel to cause the external source to enhance future data inputs into the input channel; and
receive an enhanced data input from the external source, wherein the enhanced data input was generated using the machine learning analysis output.
11 . The system of claim 10 , wherein the processor generates the machine learning analysis output, the processor is further configured to:
generate a descriptive analytics output in response to a first data field of the data input that corresponds to the format of a data structure;
assess the data input via a predictive learning analysis to determine a second data field of the data input that is not included in the format of a data structure; and
generate the machine learning analysis output via the prescriptive intelligence analysis based on the descriptive analytics output and the predictive learning analysis.
12 . The system of claim 11 , where the instructions further configure the processor to:
enhance the input channel based on the machine learning analysis output; and
generate a recommendation message based on the machine learning analysis output.
13 . The system of claim 10 , wherein the input channel comprises at least one of:
an API input, a web service input, a web portal input, or a file feed input, and
wherein the data input is received using middleware.
14 . The system of claim 10 , wherein the input channel comprises a physical document input, and
wherein the data input is received via optical character recognition (OCR).
15 . The system of claim 10 , wherein the input channel comprises at least one of:
a speech input or a text input, and
wherein the data input is received via at least one of a gateway or a natural language processing (NLP) module.
16 . The system of claim 10 , wherein the input channel comprises an email input, and
wherein the data input is received via at least one of an email webhook or the NLP module.
17 . An article of manufacture including a non-transitory, tangible computer-readable storage medium having instructions stored thereon that, in response to execution by a computer-based system, cause the computer-based system to perform operations comprising:
in response to the computer-based system determining that a format of a data input from an external source is unknown based on a comparison of the format of the data input to a format of a data structure in a database, the computer-based system further performs:
generating a predicted format for the data input using one or more machine learning models that have been trained on the data structure;
reformatting the data input to generate a reformatted data input based on the predicted format, wherein the reformatted data input comprises a standardized format comprising data fields of a specific data input type in a defined order;
storing the reformatted data input in the database;
training each of the one or more machine learning models with the reformatted data input;
moving one or more data fields of the data input to comply with the standardized format;
generating a machine learning analysis output;
in response to the machine learning analysis output, transmitting the machine learning analysis output to the external source associated with the input channel to cause the external source to enhance future data inputs into the input channel; and
receiving an enhanced data input from the external source, wherein the enhanced data input was generated using the machine learning analysis output.
18 . The article of manufacture of claim 17 , wherein the input channel comprises at least one of:
an API input, a web service input, a web portal input, or a file feed input, and
wherein the data input is received using middleware.
19 . The article of manufacture of claim 17 , wherein the input channel comprises a physical document input, and
wherein the data input is received using optical character recognition (OCR).
20 . The article of manufacture of claim 17 , wherein the input channel comprises at least one of:
a speech input or a text input, and
wherein the data input is received using at least one of a gateway or a natural language processing (NLP) module.