System and method for autonomous multi-bin parcel loading system
State of the art automated bin packing systems fail to handle dynamic scenarios in which information on dimensions of objects to be loaded is not available in advance. These systems also fail to consider capabilities of robots used for the automated packing of objects/bins. The disclosure herein generally relates to automated bin packing, and, more particularly, to a system and method for autonomous multi-bin parcel loading system. The system handles an online object packing in which information on dimensions of objects to be loaded is not available in advance. The system is also configured to consider capabilities of one or more robots used for loading objects to containers, while generating recommendations for object packing.
1. A processor implemented method for packing an object, comprising:
obtaining a current state of a first Long Distance Container (LDC) of a plurality of first Long Distance containers (LDC) in which the object is to be packed, via one or more hardware processors;
measuring, via one or more sensors coupled with the one or more hardware processors, dimensions of the object to be packed, wherein the object to be packed lies on a front line of a conveyor belt;
determining a plurality of locations in the first LDC where the object can be placed with respect to different orientations of the object, via the one or more hardware processors;
processing, using a Deep Q network (DQN), based on the current state of the first LDC;
determining, based on the processed Deep Q network (DQN), a plurality of future states of the first LDC, for placement of the object in the plurality of the determined locations for the different orientations of the object, wherein the plurality of future states are identified for each location-orientation pair, via the one or more hardware processors;
determining whether any of the plurality of future states is a state that satisfies at least one pre-defined condition with respect to packing of the objects, wherein the pre-defined condition is an expected storage space utilization resulting from the placement of the object in the plurality of determined locations for the varied orientations of the object, while considering characteristics of objects yet to be placed, via the one or more hardware processors, wherein the expected storage space utilization is determined using a Q-learning algorithm;
generating, based on the determined expected storage space utilization, object packing recommendations to a robotic arm using a trained reinforcement learning (RL) data model by determining orientation of the object while placing in the first LDC and location where the object yet to be placed in the first LDC, wherein the determined orientation and location are associated with the future state that has been determined as satisfying the at least one pre-defined condition, via the one or more hardware processors,
wherein the step of training the RL data model using a terminal reward function (r t ) that represented as
r t =ρ N-t ζ
wherein N is a total numb r of containers,
ζ is packing fraction for the whole sequence by assuming that a number of occupied containers of the total number of containers at end of the each object packing episode (T),
wherein training of the RL model with a mean squared error loss with respect to a relation:
Q ( s t ,a t )=(1−γ) r t +γQ ( s t+1 ,a t+1 ) (1)
where, s t is the current state, a t is a current action, s t+1 is a next state in which RL agent transitions, and a t+1 is a next action,
where, left-hand side part of (1) is a network output and a right-hand side part of (1) is target value, produced by a target network;
loading, by the robotic arm based on the object packing recommendations, the object to the first LDC, via the one or more hardware processors;
selecting a second LDC when none of the plurality of future states determined for the first LDC is satisfying the pre-defined condition, via the one or more hardware processors;
obtaining feedback data for the object packing recommendations after loading each object to the first LDC based on the determined orientation and location when deployed in real-world scenarios, wherein when the objects collide inside the containers during placement; and
fine-tuning the object packing recommendations with respect to placement of the object in a particular location and the orientation of the object, based on the obtained feedback data and the trained RL data model to avoid an object collision by increasing a grid size, leaving space between the containers, and updating the method with millimeter-level accurate real-time measurements.
2. The method as claimed in claim 1 , wherein determining the plurality of locations in the first LDC where the object yet to be placed comprises determining all the locations in the first LDC where the object fit in, by considering a) dimension feasibility of the object, b) capability of the robotic arm that is being used for loading the object to the first LDC, and c) stability of a stack of objects in the first LDC, for the different orientations of the object.
3. The method as claimed in claim 1 , wherein the method further comprises recurringly performing the repeating steps of opening a new the second LDC and determining the orientation and the location in the second LDC where the object yet to can be placed, if none of the plurality of future states determined for the first LDC satisfies the pre-defined condition, till the future state satisfying the pre-defined condition is identified.
4. A system for packing an object, comprising:
one or more hardware processors;
one or more sensors and a robotic arm coupled with the one or more hardware processors;
a communication interface; and
a memory storing a plurality of instructions, wherein the plurality of instructions when executed, cause the one or more hardware processors to:
obtain a current state of a first Long Distance Container (LDC) of a plurality of first Long Distance containers (LDC) in which the object is to be packed;
measure, via one or more sensors, dimensions of the object to be packed, wherein the object to be packed lies on a front line of a conveyor belt;
determine a plurality of locations in the first LDC where the object can be placed with respect to different orientations of the object;
process, using a Deep Q network (DQN), based on the current state of the first LDC;
determine, based on the processed Deep Q network (DQN), a plurality of future states of the first LDC, for placement of the object in the plurality of the determined locations for the different orientations of the object, wherein the plurality of future states are identified for each location-orientation pair;
determine whether any of the plurality of future states is a state that satisfies at least one pre-defined condition with respect to packing of the objects, wherein the pre-defined condition is an expected storage space utilization resulting from the placement of the object in the plurality of determined locations for the varied orientations of the object, while considering characteristics of objects yet to be placed, wherein the expected storage space utilization is determined using a Q-learning algorithm;
generate, based on the determined expected storage space utilization, an object packing recommendations to the robotic arm using a trained reinforcement learning (RL) data model by determining orientation of the object while placing in the first LDC and location where the object yet to be placed in the first LDC, wherein the determined orientation and location are associated with the future state that has been determined as satisfying the at least one pre-defined condition,
wherein the step of training the data model using a terminal reward function (r t ) that represented as
r t =ρ N-t ζ
wherein N is a total number of containers,
ζ is packing fraction for the whole sequence by assuming that a number of occupied containers of the total number of containers at end of the each object packing episode (T),
wherein training of the RL model with a mean squared error loss with respect to a relation:
Q ( s t ,a t )=(1−γ) r t +γQ ( s t+1 ,a t+1 ) (1)
where, s t is the current state, a t is a current action, s t+1 is a next state in which RL agent transitions, and a t+1 is a next action,
where, left-hand side part of (1) is a network output and a right-hand side part of (1) is target value, produced by a target network;
load, by the robotic arm based on the object packing recommendations, the object to the first LDC;
open a select a second LDC and determines the orientation and location in the second LDC where the object can be place, if none of the plurality of future states determined for the first LDC is satisfying the pre-defined condition;
obtain feedback data for the object packing recommendations after loading each object to the first LDC based on the determined orientation and location when deployed in real-world scenarios, wherein when the objects collide inside the containers during placement; and
fine-tune the object packing recommendations with respect to placement of the object in a particular location and the orientation of the object, based on the obtained feedback data and the trained RL data model to avoid an object collision by increasing a grid size, leaving space between the containers, and updating the method with millimeter-level accurate real-time measurements.
5. The system as claimed in claim 4 , wherein the system determines the plurality of locations in the first LDC where the object yet to be placed by determining all the locations in the first LDC where the object fit in, for the different orientations of the object.
6. A computer program product comprising a non-transitory computer readable medium having a computer readable instructions embodied therein, wherein the computer readable instructions, when executed, cause to perform packing of an object by:
obtaining a current state of a first Long Distance Container (LDC) of a plurality of first Long Distance containers (LDC) in which the object is to be packed, via one or more hardware processors;
measuring, via one or more sensors coupled with the one or more hardware processors, dimensions of the object to be packed, wherein the object to be packed lies on a front line of a conveyor belt;
determining a plurality of locations in the first LDC where the object can be placed with respect to different orientations of the object, via the one or more hardware processors;
processing, using a Deep Q network (DQN), based on the current state of the first LDC;
determining, based on the processed Deep Q network (DQN), a plurality of future states of the first LDC, for placement of the object in the plurality of the determined locations for the different orientations of the object, wherein the plurality of future states are identified for each location-orientation pair, via the one or more hardware processors;
determining whether any of the plurality of future states is a state that satisfies at least one pre-defined condition with respect to packing of the objects, wherein the pre-defined condition is an expected storage space utilization resulting from the placement of the object in the plurality of determined locations for the varied orientations of the object, while considering characteristics of objects yet to be placed, via the one or more hardware processors, wherein the expected storage space utilization is determined using a Q-learning algorithm;
generating, based on the determined expected storage space utilization, an object packing recommendations to a robotic arm using a trained reinforcement learning (RL) data model by determining orientation of the object while placing in the first LDC and location where the object yet to be placed in the first LDC, wherein the determined orientation and location are associated with the future state that has been determined as satisfying the at least one pre-defined condition, via the one or more hardware processors,
wherein the step of training the data model using a terminal reward function (r t ) that represented as
r t =ρ N-t ζ
wherein N is a total number of containers,
ζ is packing fraction for the whole sequence by assuming that a number of occupied containers of the total number of containers at end of the each object packing episode (T),
wherein training of the RL model with a mean squared error loss with respect to a relation:
Q ( s t ,a t )=(1−γ) r t +γQ ( s t+1 ,a t+1 ) (1)
where, s t is the current state, a t is a current action, s t+1 is a next state in which RL agent transitions, and a t+1 is a next action,
where, left-hand side part of (1) is a network output and a right-hand side part of (1) is target value, produced by a target network;
loading, by the robotic arm based on the object packing recommendations, the object to the first LDC, via the one or more hardware processors;
selecting a second LDC when none of the plurality of future states determined for the first LDC is satisfying the pre-defined condition, via the one or more hardware processors;
obtaining feedback data for the object packing recommendations after loading each object to the first LDC based on the determined orientation and location when deployed in real-world scenarios, wherein when the objects collide inside the containers during placement; and
fine-tuning the object packing recommendations with respect to placement of the object in a particular location and the orientation of the object, based on the obtained feedback data and the trained RL data model to avoid an object collision by increasing a grid size, leaving space between the containers, and updating the method with millimeter-level accurate real-time measurements.
7. The computer program product as claimed in claim 6 , wherein determining the plurality of locations in the first LDC where the object yet to be placed comprises determining all the locations in the first LDC where the object fit in, by considering a) dimension feasibility of the object, b) capability of the robotic arm that is being used for loading the object to the first LDC, and c) stability of a stack of objects in the first LDC, for the different orientations of the object.
8. The computer program product as claimed in claim 6 , wherein bin packing further comprises recurringly performing the repeating steps of opening a new the second LDC and determining the orientation and the location in the second LDC where the object can yet to be placed, if none of the plurality of future states determined for the first LDC satisfies the pre-defined condition, till the future state satisfying the pre-defined condition is identified.