Method and system for package movement visibility in warehouse operations

1 . A method for package movement visibility in warehouse operations by a package management system ( 1000 ), wherein the method comprises:

identifying, by the package management system ( 1000 ), at least one object entering an area of effect (AOE) and a moving in a predetermined direction, wherein the package management system ( 1000 ) determines when the at least one object approaches a region of interest (ROI) or the AOE, ascertains whether the at least one object is moving towards or away from the ROI, and, in response to determining that the at least one object is moving towards the ROI, triggers an image sensor ( 200 ) to start capturing the at least one image frame of the at least one object;

recording, by the package management system ( 1000 ), at least one image frame of the at least one object;

determining, by the package management system ( 1000 ), whether the at least one object in the at least one image frame is at least one package;

in response to determining that the at least one object in the at least one image frame is the at least one package, processing, by the package management system ( 1000 ), the at least one image frame of the at least one package in a streaming fashion as the at least one image frame becomes available;

determining, by the package management system ( 1000 ), at least one label on the at least one package from the at least one image frame;

determining, by the package management system ( 1000 ), whether the at least one label includes information associated with the at least one package;

extracting, by the package management system ( 1000 ), the information associated with the at least one package from the at least one label, in response to determining that the at least one label includes the information associated with the at least one package;

sending, by the package management system ( 1000 ), the extracted information associated with the at least one package to the cloud platform ( 400 );

determining, by the package management system ( 1000 ), a match to the extracted information associated with the at least one package in the cloud platform ( 400 ) by verifying the extracted information with information stored in a shipment database ( 440 ) mirroring an operational enterprise resource planning system ( 500 ); and

sending, by the package management system ( 1000 ), tracking details associated with the at least one package based on the match, to a client device in real-time.

2 . The method as claimed in claim 1 , wherein identifying, by the package management system ( 1000 ), the at least one object entering the AOE and moving in the predetermined direction comprises:

determining, by the package management system ( 1000 ), the at least one object approaching the package management system ( 1000 );

determining, by the package management system ( 1000 ), a proximity of the at least one object approaching the package management system ( 1000 );

determining, by the package management system ( 1000 ), that the proximity of the at least one object approaching the package management system ( 1000 ) meets a proximity threshold; and

identifying, by the package management system ( 1000 ), the at least one object is in the AOE and moving in the predetermined direction based on the proximity threshold, wherein the package management system ( 1000 ) uses a set of two sensor fusions ( 100 ) placed at different locations and directional to determine movement of the package in a specific direction and to determine whether the package is leaving or arriving with respect to the package management system ( 1000 ), the sensor fusion ( 100 ) including a multitude of sensors in combination, including IR light curtain, 2D or 3D lidars, and wherein the image sensor ( 200 ) is a 4 k resolution image sensor pointed towards the AOE and parallel to the ROI, includes an auto zoom option that is programmatically triggered such that packages of different heights are accommodated, and is placed facing downwards at 8 to 10 ft height above the ground surface.

3 . The method as claimed in claim 1 , wherein determining, by the package management system ( 1000 ), that the at least one object in the at least one image frame is at least one package comprises:

processing, by the package management system ( 1000 ), the at least one image frame using a KNN classifier created on a standard python tool called scikit-learn and trained on general processors including a CPU, such that no graphical processing unit (GPU) is needed and false negatives during identification of the package are minimized; and

determining, by the package management system ( 1000 ), that the at least one object in the captured at least one image frame is the at least one package.

4 . The method as claimed in claim 3 , further comprising:

determining, by the package management system ( 1000 ), that the at least one object in the captured at least one image frame is not the at least one package; and

stopping, by the package management system ( 1000 ), recording any further image/video of the at least one object and resetting the image sensor ( 200 ).

5 . The method as claimed in claim 4 , wherein the label detector ( 340 ), if no contours within given constraints are obtained for a given thresholding value, analyzes a next image, restricts contours by applying a sizing mechanism based on an expected minimum size and an expected maximum size for labels, uses perimeter approximation algorithms so that only contours that fall within the expected minimum size and the expected maximum size for labels are determined, uses non-max suppression when the number of contours needs to be restricted, uses OR-gate ensembling for highly permissive applications, and eliminates duplicates using non-max suppression.

6 . The method as claimed in claim 1 , wherein determining, by the package management system ( 1000 ), the match to the at least one label in the cloud platform ( 400 ) comprises:

pre-processing, by the package management system ( 1000 ), the at least one image frame and using a contour detection algorithm to find a label ROI;

detecting, by the package management system ( 1000 ), the at least one label by thresholding the at least one image frame at various values until contours of expected size and shape are obtained;

if no contours within given constraints are obtained for a given thresholding value, analyzing, by the package management system ( 1000 ), a next image and restricting the contours by applying a sizing mechanism based on an expected minimum and maximum size for labels, perimeter approximation algorithms, non-max suppression, and OR-gate ensembling;

determining, by the package management system ( 1000 ), whether the at least one label on the at least one package comprises information associated with the at least one package;

extracting, by the package management system ( 1000 ), the information associated with the at least one package from the at least one label, in response to determining that the at least one label on the at least one package comprises the information associated with the at least one package;

sending, by the package management system ( 1000 ), the extracted information associated with the at least one package from the at least one label to the cloud platform ( 400 ); and

determining, by the package management system ( 1000 ), the match to the extracted information associated with the at least one package in the cloud platform ( 400 ) by verifying the extracted information with information stored in a shipment database ( 440 ) mirroring an operational enterprise resource planning system ( 500 ).

7 . The method as claimed in claim 6 , further comprising:

determining, by the package management system ( 1000 ), that the at least one label on the at least one package does not have information/barcode or that the information/barcode detected on the at least one label does not have a match in the cloud platform ( 400 );

storing, by the package management system ( 1000 ), the at least one image frame of the at least one label at an edge computing device ( 300 ) of the package management system ( 1000 ) and storing details related to the at least one package in a label database ( 460 ) of the cloud platform ( 400 ); and

sending, by the package management system ( 1000 ), the at least one image frame of the at least one label to the cloud platform ( 400 ) for initiating a potential manual intervention.

8 . The method as claimed in claim 1 , wherein the at least one label on the at least one package comprises at least one of barcodes, text, contextual information associated with the at least one package, instructions for handling the at least one package and hazard warning associated with the at least one package.

9 . The method as claimed in claim 1 , wherein the tracking details associated with the at least one package comprise at least one of a timestamp of inbound and outbound and location information associated with the at least one package, and wherein once the match is found the timestamp of inbound and outbound and the location information are sent to the operational enterprise resource planning system ( 500 ).

10 . The method as claimed in claim 1 , wherein the package management system ( 1000 ) operates in offline mode such that barcode data is processed and stored at the edge computing device ( 300 ) locally and, once network availability is restored and the package management system ( 1000 ) is online, the barcode data stored at the edge computing device ( 300 ) locally is fed to the cloud platform ( 400 ).

11 . A package management system ( 1000 ) for package movement visibility in warehouse operations, wherein the package management system ( 1000 ) comprises:

a set of sensor fusion ( 100 ) configured to identify at least one object entering an area of effect (AOE) and moving in a predetermined direction by determining when the at least one object approaches a region of interest (ROI) or the AOE, ascertaining whether the at least one object is moving towards or away from the ROI, and, in response to determining that the at least one object is moving towards the ROI, triggering at least one image sensor ( 200 ) to start capturing at least one image frame of the at least one object;

at least one image sensor ( 200 ) configured to record at least one image frame of the at least one object;

an edge computing device ( 300 ) comprising a package identifier ( 320 ), a label detector ( 340 ), and an information extractor ( 360 ), the edge computing device ( 300 ) configured to determine whether the at least one object in the at least one image frame is at least one package, in response to a determination that the at least one object in the at least one image frame is the at least one package process the at least one image frame in a streaming fashion as the at least one image frame becomes available, determine at least one label on the at least one package from the at least one image frame, determine whether the at least one label includes information associated with the at least one package, extract the information associated with the at least one package from the at least one label, and send the extracted information associated with the at least one package to a cloud platform ( 400 );

cloud

a cloud platform ( 400 ) comprising a label identification module ( 420 ) and a shipment database ( 440 ) mirroring an operational enterprise resource planning system ( 500 ), wherein the cloud platform ( 400 ) configured to verify the extracted information with information stored in the shipment database ( 440 ), determine a match to the extracted information associated with the at least one package, and send tracking details associated with the at least one package based on the match to a client device in real-time.

12 . The package management system ( 1000 ) as claimed in claim 11 , wherein identifying the at least one object entering the AOE and moving in the predetermined direction by the set of sensor fusion ( 100 ) comprises:

determine the at least one object approaching the package management system ( 1000 );

determine a proximity of the at least one object approaching the package management system ( 1000 );

determine that the proximity of the at least one object approaching the package management system ( 1000 ) meets a proximity threshold; and

identify the at least one object is in the AOE and moving in the predetermined direction based on the proximity threshold, wherein the package management system ( 1000 ) comprises a set of two sensor fusions ( 100 ) placed at different locations, the set of two sensor fusions ( 100 ) being directional to determine movement of the package in a specific direction and to determine whether the package is leaving or arriving with respect to the package management system ( 1000 ), the sensor fusion ( 100 ) including a multitude of sensors in combination, including IR light curtain, 2D or 3D lidars, and wherein the at least one image sensor ( 200 ) is a 4 k resolution image sensor pointed towards the AOE and parallel to the ROI, includes an auto zoom option that is programmatically triggered such that packages of different heights are accommodated, and is placed facing downwards at 8 to 10 ft height above the ground surface.

13 . The package management system ( 1000 ) as claimed in claim 11 , wherein determining that the at least one object in the at least one image frame is at least one package by the edge computing device ( 300 ) comprises:

process the at least one image frame using a KNN classifier created on a standard python tool called scikit-learn and trained on general processors including a CPU, such that no graphical processing unit (GPU) is needed and false negatives during identification of the package by the edge computing device ( 300 ) are minimized; and

determine that the at least one object in the captured at least one image frame is the at least one package.

14 . The package management system ( 1000 ) as claimed in claim 13 , wherein the edge computing device ( 300 ) is further configured:

determine that the at least one object in the captured at least one image frame is not the at least one package; and

stop recording any further image/video of the at least one object and reset the at least one image sensor ( 200 ).

15 . The package management system ( 1000 ) as claimed in claim 14 , wherein the label detector ( 340 ) is further configured, if no contours within given constraints are obtained for a given thresholding value, to analyze a next image, to restrict contours by applying a sizing mechanism based on an expected minimum size and an expected maximum size for labels, to use perimeter approximation algorithms so that only contours that fall within the expected minimum size and the expected maximum size for labels are determined, to use non-max suppression when the number of contours needs to be restricted, to use OR-gate ensembling for highly permissive applications, and to eliminate duplicates using non-max suppression.

16 . The package management system ( 1000 ) as claimed in claim 11 , wherein determining the match to the at least one label in the cloud platform ( 400 ) comprises:

pre-process the at least one image frame and use a contour detection algorithm by the label detector ( 340 ) to find a label ROI;

detect, by the label detector ( 340 ), the at least one label by thresholding the at least one image frame at various values until contours of expected size and shape are obtained;

if no contours within given constraints are obtained for a given thresholding value, analyze a next image and restrict the contours by applying a sizing mechanism based on an expected minimum and maximum size for labels, perimeter approximation algorithms, non-max suppression, and OR-gate ensembling;

determine by the edge computing device ( 300 ) whether the at least one label on the at least one package comprises information associated with the at least one package;

extract by the edge computing device ( 300 ) the information associated with the at least one package from the at least one label, in response to determining that the at least one label on the at least one package comprises the information associated with the at least one package;

send by the edge computing device ( 300 ) the extracted information associated with the at least one package from the at least one label to the cloud platform ( 400 ); and

determine by the cloud platform ( 400 ) the match to the extracted information associated with the at least one package in the cloud platform ( 400 ) by verifying the extracted information with information stored in a shipment database ( 440 ) mirroring an operational enterprise resource planning system ( 500 ) and using a matching algorithm to match scanned barcodes, extracted text, and other contextual information from the label with the information in the shipment database ( 440 ).

17 . The package management system ( 1000 ) as claimed in claim 16 , wherein the edge computing device ( 300 ) is further configured to:

determine that the at least one label on the at least one package does not comprise the information associated with the at least one package or that the information/barcode detected on the at least one label does not have a match in the cloud platform ( 400 );

store the at least one image frame of the at least one label and store details related to the at least one package in a label database ( 460 ) of the cloud platform ( 400 ); and

send the at least one image frame of the at least one label to the cloud platform ( 400 ) for initiating potential manual intervention.

18 . The package management system ( 1000 ) as claimed in claim 11 , wherein the at least one label on the at least one package comprises at least one of barcodes, text, contextual information associated with the at least one package, instructions for handling the at least one package and hazard warning associated with the at least one package.

19 . The package management system ( 1000 ) as claimed in claim 11 , wherein the tracking details associated with the at least one package comprise at least one of a timestamp of inbound and outbound and location information associated with the at least one package, and wherein once the match is found the timestamp of inbound and outbound and the location information are sent to the operational enterprise resource planning system ( 500 ).

20 . The package management system ( 1000 ) as claimed in claim 11 , wherein the package management system ( 1000 ) is configured to operate in offline mode such that barcode data is processed and stored at the edge computing device ( 300 ) locally and, once network availability is restored and the package management system ( 1000 ) is online, the barcode data stored at the edge computing device ( 300 ) locally is fed to the cloud platform ( 400 ).