Systems and methods for data ingestion for supply chain optimization
The present invention relates to systems and methods for ingesting of raw client data into a supply chain optimization system. The client enterprise data system (EDS) provides raw data, typically in tabular format across many files, to a data management module of the optimization system. This data is then profiled. and a set of mapping AI models is applied to the profiled data. Each of the plurality of mapping AI models corresponds to a single input feature of the optimization model. As the mapping models are applied, a live preview is generated. This live preview is reviewable by a human operator, and human input can be provided. When there is input, the individual mapping model is updated (without impacting the other mapping models), and a new live preview for that given feature is generated. The resulting standardized feature set may be consumed by the supply chain optimization model.
1 . A computerized method for data ingestion from a client enterprise data system (EDS) for consumption by a supply chain optimization model, the method comprising:
receiving raw data in a plurality of data files from a remote EDS, wherein the plurality of data files includes a plurality of tables, and wherein the raw data is non-standardized;
storing the raw data;
disambiguating headers of the plurality of tables by:
initially performing a keyword search and conceptual clustering to standardize headers;
when the keyword search and conceptual clustering fails then applying a three-pronged advanced disambiguation process of locating physically adjacent words in the table that provide header context, comparison of table file name to the header, and clustering of values in a column below the header against known values and applying a conceptual lexicographic chart to relate the known values to a category;
applying a plurality of mapping artificial intelligence (AI) models to the disambiguated plurality of tables, wherein each of the plurality of mapping AI models corresponds to a single input in a destination data system to generate a standardized destination data system;
generating a live preview of an output for each mapping AI model as it is applied; and
transmitting the generated standardized destination data system for at least one downstream application.
2 . The method of claim 1 , further comprising normalizing the raw data.
3 . The method of claim 2 , wherein the normalizing includes imputing values into the plurality of tables.
4 . The method of claim 3 , where the imputing includes determining if a null value is a true zero, an error, or non-data.
5 . The method of claim 1 , further comprising determining a sensitivity of a supply chain optimization model to any given value in the raw data.
6 . The method of claim 1 , further comprising receiving human intervention after the live preview is generated.
7 . The method of claim 6 , further comprising iteratively altering at least one of the plurality of mapping AI models in response to the human intervention.
8 . The method of claim 1 , wherein the standardized is one of user defined standard, third-party standard, or default standard.
9 . The method of claim 1 , further comprising
optimizing a supply chain by applying a supply chain optimization model to the standardized destination data system;
determining the most impactful changes to the supply chain responsive to the optimization; and
transmitting the most impactful changes to the remote EDS.
10 . A computerized system for data ingestion from a client enterprise data system (EDS) for consumption by a supply chain optimization model, the system comprising:
a network connection for receiving raw data in a plurality of data files from a remote EDS, wherein the plurality of data files includes a plurality of tables, and wherein the raw data is non-standardized;
a database for storing the raw data;
a server for executing computer code to:
disambiguate headers of the plurality of tables by:
initially performing a keyword search and conceptual clustering to standardize headers;
when the keyword search and conceptual clustering fails then applying a three-pronged advanced disambiguation process of locating physically adjacent words in the table that provide header context, comparison of table file name to the header, and clustering of values in a column below the header against known values and applying a conceptual lexicographic chart to relate the known values to a category;
apply a plurality of mapping artificial intelligence (AI) models to the disambiguated plurality of tables, wherein each of the plurality of mapping AI models corresponds to a single input of a destination data system of a supply chain optimization model to generate a standardized destination data system;
generate a live preview of an output for each mapping AI model as it is applied; and
transmit the generated standardized destination data system for at least one downstream application.
11 . The system of claim 10 , wherein the server further normalizes the raw data.
12 . The system of claim 11 , wherein the normalizing includes imputing values into the plurality of tables.
13 . The system of claim 12 , where the imputing includes determining if a null value is a true zero, an error, or non-data.
14 . The system of claim 10 , wherein the server further determines a sensitivity of the supply chain optimization model to any given value in the raw data.
15 . The system of claim 10 , further comprising an interface for receiving human intervention after the live preview is generated.
16 . The system of claim 15 , wherein the server further iteratively alters at least one of the plurality of mapping AI models in response to the human intervention.
17 . The system of claim 10 , wherein the standardized is one of user defined standard, third party standard, or default standard.
18 . The system of claim 10 , further comprising a second server configured to perform the steps of:
optimizing a supply chain by applying a supply chain optimization model to the standardized feature set;
determining the most impactful changes to the supply chain responsive to the optimization; and
transmitting the most impactful changes to the remote EDS.