System and method for supporting execution of batch production using reinforcement learning
A computer-implemented method for supporting execution of batch production by a production system includes, over a sequence of steps: acquiring a system state defined by a shop floor status, inventory status and a demand of the product types, and processing the system state using a reinforcement learned policy including a deep learning model to output a control action defining an integer batch size of a selected product type. The control action is determined by using learned parameters of the deep learning model to compute logits for a categorical distribution of predicted product types and a categorical distribution of predicted batch sizes from the system state. The logits are processed to transform the categorical distribution of predicted product types into an encoding of the selected product type and reduce the categorical distribution of predicted batch sizes into an integer batch size, for producing a next batch on the shop floor.
1 . A computer-implemented method for supporting execution of batch production by a production system comprising production machines configured to produce a number of different product types on a shop floor, the method comprising:
over a sequence of steps, iteratively performing:
acquiring a system state defined by a shop floor status, inventory status and a product demand of the product types,
processing the system state using a reinforcement learned policy comprising a deep learning model to output a control action defining an integer batch size of a selected product type, wherein the control action is determined by:
using learned parameters of the deep learning model to compute logits for a categorical distribution of predicted product types and a categorical distribution of predicted batch sizes from the system state, and
processing the logits to transform the categorical distribution of predicted product types into an encoding of the selected product type and reduce the categorical distribution of predicted batch sizes into an integer batch size, and
communicating the control action to the production machines to effect production of a next batch on the shop floor based on the integer batch size of the selected product type.
2 . The method according to claim 1 , wherein the shop floor status defines a utilization of resources on the shop floor by one or more current batches in production, which is indicative of a remaining production time of the one or more current batches.
3 . The method according to claim 1 , wherein the policy is trained in a reinforcement learning process that comprises, over a sequence of steps, optimizing learnable parameters of the policy to maximize a cumulative reward resulting from a sequence of control actions, based on a reward function defined based on a total production cost.
4 . The method according to claim 3 , wherein the total production cost includes cost of raw materials, cost of secondary resources and cost of obsolescence of products in inventory.
5 . The method according to claim 1 , wherein the processing of the logits comprises:
creating respective discretized vector representations of the predicted product types and the predicted batch sizes based on the respective logits, wherein the discretized vector representation of the predicted product types defines the encoding of selected product type, and
determining the integer batch size from the discretized vector representation of the predicted batch sizes using a linear transformation.
6 . The method according to claim 5 , wherein the respective discretized vector representations are created by:
perturbing the logits for the categorical distributions of the predicted product types and the predicted batch sizes with random noise to create biased samples that represent differentiable approximations of samples of the respective categorical distributions, and
computing a first one-hot vector encoding of the biased samples corresponding to the predicted product types and a second one-hot vector encoding of the biased samples corresponding to the predicted batch sizes,
wherein the first one-hot vector encoding defines the encoding of the selected product type.
7 . The method according to claim 6 , wherein the biased samples are created using a Gumbel-Softmax estimator.
8 . The method according to claim 6 , wherein the linear transformation comprises an inner product of the second one-hot vector encoding and the vector [1, . . . n], where n denotes a dimensionality of the second one-hot vector encoding defined by the number of possible batch sizes.
9 . The method according to claim 1 , wherein the control action is determined based on a scalar multiplication of the encoding of the selected product type by the integer batch size.
10 . A computing system for supporting execution of batch production by a production system comprising production machines configured to produce a number of different product types on a shop floor, the system comprising:
one or more processors, and
memory storing algorithmic modules executable by the one or more processors, the algorithmic modules comprising:
a production execution interface module configured to acquire a system state defined by a shop floor status, inventory status and a product demand of the product types, and
a batch size optimization module configured to process the system state using a reinforcement learned policy comprising a deep learning model to output a control action defining an integer batch size of a selected product type, wherein the control action is determined by:
using learned parameters of the deep learning model to compute logits for a categorical distribution of predicted product types and a categorical distribution of predicted batch sizes from the system state, and
processing the logits to transform the categorical distribution of predicted product types into an encoding of the selected product type and reduce the categorical distribution of predicted batch sizes into an integer batch size,
wherein the production execution interface module is configured to communicate the control action to the production system-machines to effect production of a next batch on the shop floor based on the integer batch size of the selected product type, and
wherein the algorithmic modules are iteratively executable over a sequence of steps to support batch production by the production system.