System And Method For Reinforcement-Learning Based On-Loading Optimization
A method and system are provided where a module employing reinforcement learning (RL) can learn to solve the vehicle selection and space allocation problems for the transportation and/or storage of goods. In one embodiment, a method includes: (a) obtaining a specification of a load that includes enclosures of one or more enclosure types, and the specification includes, for each enclosure type: (i) dimensions of an enclosure of the enclosure type, and (ii) a number of enclosures of the enclosure type. The method also includes (b) obtaining a specification of vehicles of one or more vehicle types, where the specification includes, for each vehicle type: (i) dimensions of space available within a vehicle of the vehicle type, and (ii) a number of vehicles of the vehicle type that are available for transportation. The method further includes (c) providing a simulation environment for simulating loading of a vehicle. In addition, the method includes (d) selecting by an agent module a vehicle of a particular type, where: the selected vehicle has space available to accommodate at least a portion of the load; and the selection is based on a state of the environment, an observation of the environment, and a reward received previously from the environment; (e) receiving by the agent module a current reward from the environment in response to selecting the vehicle; and (f) repeating by the agent module steps (d) and (e) until the load is accommodated within space available in the one or more selected vehicles.
1 . A method for selecting one or more vehicles for transporting goods, the method comprising:
(a) obtaining a specification of a load comprising enclosures of one or more enclosure types, the specification comprising, for each enclosure type: (i) dimensions of an enclosure of the enclosure type, and (ii) a number of enclosures of the enclosure type;
(b) obtaining a specification of vehicles of one or more vehicle types, the specification comprising, for each vehicle type: (i) dimensions of space available within a vehicle of the vehicle type, and (ii) a number of vehicles of the vehicle type that are available for transportation;
(c) providing a simulation environment for simulating loading of a vehicle;
(d) selecting by an agent module a vehicle of a particular type, wherein:
the selected vehicle has space available to accommodate at least a portion of the load; and
the selection is based on a state of the environment, an observation of the environment, and a reward received previously from the environment;
(e) receiving by the agent module a current reward from the environment in response to selecting the vehicle; and
(f) repeating by the agent module steps (d) and (e) until the load is accommodated within space available in the one or more selected vehicles.
2 . The method of claim 1 , wherein the specification of the load comprises one or more of:
a weight of a particular enclosure from the load; or
a constraint on an orientation of the particular enclosure.
3 . The method of claim 1 , wherein the specification of vehicles comprises one or more of:
a weight capacity of a particular vehicle; or
a constraint on routes taken by the particular vehicle.
4 . The method of claim 1 , wherein:
the state comprises one or more types of vehicles available and respective numbers of vehicles that are available of each type;
the observation comprises one or more types of enclosures that remain to be loaded on to one or more vehicles, and respective numbers of enclosures of each such type; and
the reward is derived from cost of the selected vehicle.
5 . The method of claim 1 , wherein the agent comprises a Q-learning module or an artificial neural network (ANN).
6 . A method for on-loading a vehicle for transporting goods, the method comprising:
(a) obtaining a specification of a load comprising enclosures of one or more enclosure types, the specification comprising, for each enclosure type: (i) dimensions of an enclosure of the enclosure type, and (ii) a number of enclosures of the enclosure type;
(b) obtaining a vehicle specification comprising a set of dimensions representing one or more spaces within the vehicle that are available for on-loading;
(c) providing a simulation environment for simulating loading of the vehicle;
(d) selecting by an agent module a location within the one more spaces for placement of an enclosure chosen from the load for placement, wherein the selection of the location is based on a state of the environment, an observation of the environment, a reward received previously from the environment, and dimensions of the chosen enclosure;
(e) receiving by the agent module a current reward from the environment in response to selecting the location for the placement of the chosen enclosure; and
(f) repeating by the agent module steps (d) and (e) until the load is accommodated within the one or more spaces available within the vehicle.
7 . The method of claim 6 , wherein the specification of the load comprises one or more of:
a weight of a particular enclosure from the load; or
a constraint on an orientation of the particular enclosure.
8 . The method of claim 6 , wherein the vehicle specification comprises one or more of:
a weight capacity of the vehicle; or
a constraint on routes taken by the vehicle.
9 . The method of claim 6 , further comprising:
selecting by the agent module a new orientation of the chosen enclosure.
10 . The method of claim 6 , wherein:
the state comprises an identification of one or more spaces within the vehicle that are occupied;
the observation comprises one or more types of enclosures that remain to be loaded on to the vehicle, and respective numbers of enclosures of each such type; and
the reward is derived from a change in volume of the one or more spaces within the vehicle that are available for on-loading resulting from placement of the chosen enclosure within the one or more spaces.
11 . The method of claim 6 , wherein the reward is derived from a change in weigh of the vehicle resulting from placement of the chosen enclosure within the one or more spaces within the vehicle that are available for on-loading.
12 . The method of claim 6 , wherein the agent comprises a a-learning module or an artificial neural network (ANN).
13 . A system for selecting one or more vehicles for transporting goods, the system comprising:
a processor; and
a memory in communication with the processor and comprising instructions which, when executed by the processor, program the processor to:
(a) obtain a specification of a load comprising enclosures of one or more enclosure types, the specification comprising, for each enclosure type: (i) dimensions of an enclosure of the enclosure type, and (ii) a number of enclosures of the enclosure type;
(b) obtain a specification of vehicles of one or more vehicle types, the specification comprising, for each vehicle type: (i) dimensions of space available within a vehicle of the vehicle type, and (ii) a number of vehicles of the vehicle type that are available for transportation;
(c) provide a simulation environment for simulating loading of a vehicle; and
(d) operate as an agent module, wherein the instructions program the agent module to:
(A) select a vehicle of a particular type, wherein: (i) the selected vehicle has space available to accommodate at least a portion of the load; and (ii) the selection is based on a state of the environment, an observation of the environment and a reward received previously from the environment;
(B) receive a current reward from the environment in response to selecting the vehicle; and
(C) repeat operations (d)(A) and (d)(B) until the load is accommodated within space available in the one or more selected vehicles.
14 . The system of claim 13 , wherein the specification of the load comprises one or more of:
a weight of a particular enclosure from the load; or
a constraint on an orientation of the particular enclosure.
15 . The system of claim 13 , wherein the specification of vehicles comprises one or more of:
a weight capacity of a particular vehicle; or
a constraint on routes taken by the particular vehicle.
16 . The system of claim 13 , wherein:
the state comprises one or more types of vehicles available and respective numbers of vehicles that are available of each type;
the observation comprises one or more types of enclosures that remain to be loaded on to one or more vehicles, and respective numbers of enclosures of each such type; and
the reward is derived from cost of the selected vehicle.
17 . The system of claim 13 , wherein the agent comprises a Q-learning module or an artificial neural network (ANN).
18 . A system for on-loading a vehicle for transporting goods, the system comprising:
a processor; and
a memory in communication with the processor and comprising instructions which, when executed by the processor, program the processor to:
(a) obtain a specification of a load comprising enclosures of one or more enclosure types, the specification comprising, for each enclosure type: (i) dimensions of an enclosure of the enclosure type, and (ii) a number of enclosures of the enclosure type;
(b) obtain a vehicle specification comprising a set of dimensions representing one or more spaces within the vehicle that are available for on-loading;
(c) provide a simulation environment for simulating loading of the vehicle; and
(d) operate as an agent module, wherein the instructions program the agent module to:
(A) select a location within the one more spaces for placement of an enclosure chosen from the load for placement, wherein the selection of the location is based on a state of the environment, an observation of the environment, a reward received previously from the environment, and dimensions of the chosen enclosure;
(B) receive a current reward from the environment in response to selecting the location for the placement of the chosen enclosure; and
(C) repeat by operations (d)(A) and (d)(B) until the load is accommodated within the one or more spaces available within the vehicle.
19 . The system of claim 18 , wherein the specification of the load comprises one or more of:
a weight of a particular enclosure from the load; or
a constraint on an orientation of the particular enclosure.
20 . The system of claim 18 , wherein the vehicle specification comprises one or more of:
a weight capacity of the vehicle; or
a constraint on routes taken by the vehicle.
21 . The system of claim 18 , wherein the instructions program the agent module to select a new orientation of the chosen enclosure.
22 . The system of claim 18 , wherein:
the state comprises an identification of one or more spaces within the vehicle that are occupied;
the observation comprises one or more types of enclosures that remain to be loaded on to the vehicle, and respective numbers of enclosures of each such type; and
the reward is derived from a change in volume of the one or more spaces within the vehicle that are available for on-loading resulting from placement of the chosen enclosure within the one or more spaces.
23 . The system of claim 18 , wherein the reward is derived from a change in weigh of the vehicle resulting from placement of the chosen enclosure within the one or more spaces within the vehicle that are available for on-loading.
24 . The system of claim 18 , wherein the agent comprises a Q-learning module or an artificial neural network (ANN).