DATA INGESTION MANAGEMENT
An example computer system for ingestion of data can include: one or more processors; and non-transitory computer-readable storage media encoding instructions which, when executed by the one or more processors, causes the computer system to: authenticate a user to allow for definition of a configuration file for the ingestion of the data; receive the configuration file, with the configuration file defining parameters for the ingestion of the data; extract the data through an application programming interface according to the parameters of the configuration file; and perform remediation on an error record in the data according to an error code associated with the error record.
1 . A computer system for ingestion of data, comprising:
one or more processors; and
non-transitory computer-readable storage media encoding instructions which, when executed by the one or more processors, causes the computer system to:
authenticate a user to allow for definition of a configuration file for the ingestion of the data;
receive the configuration file, with the configuration file defining parameters for the ingestion of the data;
extract the data through an application programming interface according to the parameters of the configuration file; and
perform remediation on an error record in the data according to an error code associated with the error record.
2 . The computer system of claim 1 , wherein authentication of the user is configurable and includes generation of a token.
3 . The computer system of claim 1 , wherein the parameters of the configuration file define a number of columns and a number of slots for performing the ingestion of the data.
4 . The computer system of claim 3 , wherein the number of columns and the number of slots are configurable to control parallelization of extraction, processing, and the ingestion of the data.
5 . The computer system of claim 1 , wherein the parameters of the configuration file define a date format for the ingestion of the data.
6 . The computer system of claim 1 , comprising further instructions which, when executed by the one or more processors, causes the computer system to preprocess the data for the ingestion by dynamically requesting Uniform Resource Locator generation of the application programming interface for the ingestion.
7 . The computer system of claim 1 , comprising further instructions which, when executed by the one or more processors, causes the computer system to postprocess the data by transforming the data after the ingestion, including conversion of semi-structured data to structured data.
8 . The computer system of claim 1 , wherein the application programming interface for the ingestion of the data is a Representational State Transfer application programming interface.
9 . The computer system of claim 1 , wherein a number of tries for the remediation of the error record is configurable.
10 . The computer system of claim 1 , wherein the error code dictates a type of the remediation of the error record.
11 . A method for ingestion of data, comprising:
authenticating a user to allow for definition of a configuration file for the ingestion of the data;
receiving the configuration file, with the configuration file defining parameters for the ingestion of the data;
extracting the data through an application programming interface according to the parameters of the configuration file; and
performing remediation on an error record in the data according to an error code associated with the error record.
12 . The method of claim 11 , wherein authentication of the user is configurable and includes generation of a token.
13 . The method of claim 11 , wherein the parameters of the configuration file define a number of columns and a number of slots for performing the ingestion of the data.
14 . The method of claim 13 , wherein the number of columns and the number of slots are configurable to control parallelization of extraction, processing, and the ingestion of the data.
15 . The method of claim 11 , wherein the parameters of the configuration file define a date format for the ingestion of the data.
16 . The method of claim 11 , further comprising preprocessing the data for the ingestion by dynamically requesting Uniform Resource Locator generation of the application programming interface for the ingestion.
17 . The method of claim 11 , further comprising postprocessing the data by transforming the data after the ingestion, including conversion of semi-structured data to structured data.
18 . The method of claim 11 , wherein the application programming interface for the ingestion of the data is a Representational State Transfer application programming interface.
19 . The method of claim 11 , wherein a number of tries for the remediation of the error record is configurable.
20 . The method of claim 11 , wherein the error code dictates a type of the remediation of the error record.