Automated supply chain demand forecasting
In an embodiment, a method includes receiving training data representing historic consumer demand for products, detecting changepoints in that data that may be associated with disruptive events, identifying relevant data for modeling, performing clustering, processing configuration information, training one or more machine learning models that are capable of evaluating other received data more accurately, and outputting results to a user display device.
1 . A computer-implemented method of forecasting supply chain demand of products of goods or services, executed by a distributed computer system comprising data acquisition logic, project management logic, data storage logic managing a database storing product demand data, third-party data, training data, testing data, and output data, and data processing logic that exposes a retraining web service and a scoring web service deployed in a cloud production environment, the method comprising:
receiving, by the data acquisition logic, a training data set comprising product demand data indicating consumer demand for millions of products at a plurality of time points;
when downstream consumption data is unavailable, and only upstream demand signals are present, estimating true demand by imputing sales using a substitution-split imputation that allocates incremental demand across substitute SKUs proportionally to historical shares and/or by using a site-visit-to-sales ratio for primarily online items;
clustering the training data set into a plurality of time series clusters using K-means over normalized and smoothed demand-index windows since disruption onset to obtain cluster representatives;
executing, by the data processing logic, a supervised multi-class machine learning classifier aided by moving average convergence/divergence (MACD) indicators to output calculations of one or more changepoints representative time series for the clusters, each of the changepoints corresponding to a disruptive event;
retrieving a baseline forecast for one or more of the products calculated prior to a detected changepoint and calculating one or more of a mean demand level, a median demand level, or a standard deviation of demand level for selected periods of the training data set that are before and after the detected changepoint;
identifying a deviation between the baseline forecast and a particular time series cluster among the plurality of the time series clusters before and after the detected changepoint corresponding to a disruptive event, the deviation exceeding, 1.5*Inter Quartile Range (IQR) above the 75th percentile or below the 25th percentile of the baseline forecast, and in response, flagging the particular time series cluster as impacted by a disruptive event;
processing configuration information that specifies third-party data for training one or more machine learning models and, in response thereto, accessing one or more of mobility tracking data specifying a percent change in visits to stores within a geographic area, a social distance index, school closures data, case count data, unemployment claims data, consumer sentiment data, or hospital utilization data;
engineering non-forward-looking regressors by generating lagged lead indicators and window statistics with a starting point limited by a forecast horizon, and selecting optimal lags by correlation analysis on the lagged regressors;
transforming the training data set and the third-party data to be used for training the one or more machine learning models into a format suitable for the one or more machine learning models by one or more of: resizing inputs to a particular fixed size, converting non-numeric data features into numeric feature ones, normalizing numeric data features, or lower-casing or tokenizing metadata text features;
training new machine learning models only for the time series clusters flagged as impacted, based on the training data set and the third-party data, and validating the trained models using walk-forward cross-validation to prevent time-dependent data leakage;
applying hierarchical time-series reconciliation to propagate detected level shifts coherently between aggregated and disaggregated levels across a product and/or location hierarchy;
inputting one or more new timestamps respectively associated with one or more products; and
executing the scoring web service to generate forecasts for the new timestamps, transmitting forecasted results to a consumption client via an API, and rendering forecast visualizations on a user display device, and executing the retraining web service to retrain models for flagged series or clusters.
2 . The computer-implemented method of claim 1 , wherein at least a portion of the product demand data comprising upstream consumption data that corresponds to one or more other computing systems that are upstream from the distributed computer system; the method further comprising updating the product demand data for the one or more products by imputing sales values based on the upstream consumption data.
3 . The computer-implemented method of claim 1 , further comprising clustering the training data set into a plurality of time series clusters based on the MACD indicators that are associated with the product demand data.
4 . One or more non-transitory computer-readable storage media storing one or more sequences of instructions which, when executed using one or more processors of a distributed computer system comprising data acquisition logic, project management logic, data storage logic managing a database storing product demand data, third-party data, training data, testing data, and output data, and data processing logic that exposes a retraining web service and a scoring web service deployed in a cloud production environment, cause the one or more processors to execute:
receiving, by the data acquisition logic, a training data set comprising product demand data indicating consumer demand for millions of products at a plurality of time points;
when downstream consumption data is unavailable, and only upstream demand signals are present, estimating true demand by imputing sales using a substitution-split imputation that allocates incremental demand across substitute SKUs proportionally to historical shares and/or by using a site-visit-to-sales ratio for primarily online items;
clustering the training data set into a plurality of time series clusters using K-means over normalized and smoothed demand-index windows since disruption onset to obtain cluster representatives;
executing a supervised multi-class machine learning classifier aided by moving average convergence/divergence (MACD) indicators to output calculations of one or more changepoints representative time series for the clusters, each of the changepoints corresponding to a disruptive event;
retrieving a baseline forecast for one or more of the products calculated prior to a detected changepoint and calculating one or more of a mean demand level, a median demand level, or a standard deviation of demand level for selected periods of the training data set that are before and after the detected changepoint;
identifying a deviation between the baseline forecast and a particular time series cluster among the plurality of the time series clusters before and after the detected changepoint corresponding to a disruptive event, the deviation exceeding, 1.5*Inter Quartile Range (IQR) above the 75th percentile or below the 25th percentile of the baseline forecast, and in response, flagging the particular time series cluster as impacted by a disruptive event;
processing configuration information that specifies third-party data for training one or more machine learning models and, in response thereto, accessing one or more of mobility tracking data specifying a percent change in visits to stores within a geographic area, a social distance index, school closures data, case count data, unemployment claims data, consumer sentiment data, or hospital utilization data;
engineering non-forward-looking regressors by generating lagged lead indicators and window statistics with a starting point limited by a forecast horizon, and selecting optimal lags by correlation analysis on the lagged regressors;
transforming the training data set and the third-party data to be used for training the one or more machine learning models into a format suitable for the one or more machine learning models by one or more of: resizing inputs to a particular fixed size, converting non-numeric data features into numeric feature ones, normalizing numeric data features, or lower-casing or tokenizing metadata text features;
training new machine learning models only for the time series clusters flagged as impacted, based on the training data set and the third-party data, and validating the trained models using walk-forward cross-validation to prevent time-dependent data leakage;
applying hierarchical time-series reconciliation to propagate detected level shifts coherently between aggregated and disaggregated levels across a product and/or location hierarchy;
inputting one or more new timestamps respectively associated with one or more products; and
executing the scoring web service to generate forecasts for the new timestamps, transmitting forecasted results to a consumption client via an API, and rendering forecast visualizations on a user display device, and executing the retraining web service to retrain models for flagged series or clusters.
5 . The one or more non-transitory computer-readable storage media of claim 4 , wherein at least a portion of the product demand data comprising upstream consumption data that corresponds to one or more other computing systems that are upstream from the distributed computer system; further comprising one or more sequences of instructions which, when executed using the one or more processors, cause the one or more processors to execute: updating the product demand data for the one or more products by imputing sales values based on the upstream consumption data.
6 . The one or more non-transitory computer-readable storage media of claim 4 , further comprising one or more sequences of instructions which, when executed using the one or more processors, cause the one or more processors to execute: clustering the training data set into a plurality of time series clusters based on the MACD indicators that are associated with the product demand data.