Granted Patent B2

US 12,487,337 · App. 18/732,342 · Granted Dec 2, 2025

Inventor: Delbert A. Joseph (Cumming, GA)

Assignee: Intelligrated Headquarters, LLC

G01S7/4817B65G1/1371G01S17/06G01S17/89B25J9/163B65G2203/02B65G2203/044

Abstract

Claims (78)

1 . A method of monitoring an operation in a material handling environment, the method comprising:

generating a machine learning model by using a data stream representative of a three-dimensional (3D) scan of a target area and operational specification data associated with the operation to be performed in the material handling environment as inputs to train the machine learning model;

comparing, by using the machine learning model, a first set of data points derived from the data stream comprising a 3D point cloud of the material handling environment with a second set of data points derived from the operational specification data; and

determining, by using the machine learning model, a performance status associated with an efficiency of an execution of the operation.

2 . The method of claim 1 further comprising:

scanning, by a LiDAR based sensor, the target area of the material handling environment; and

generating the data stream comprising the 3D point cloud associated with at least one of a machine, an operator, or a section of the material handling environment.

3 . The method of claim 1 further comprising:

analyzing the data stream to identify a choke point associated with the operation in the material handling environment, wherein the choke point is indicative of a fault condition associated with an operator of at least one of a machine or a workflow performed by the operator in the material handling environment; and

generating a message indicating at least one of the choke point or a remedial action to be performed to resolve the choke point.

4 . The method of claim 1 further comprising:

determining a new key performance indicator associated with the operation of at least one of a machine, an operator, or a workflow process to be performed in the material handling environment based on analysis of the operational specification data and the data stream by the machine learning model; and

modifying the operational specification data by including the new key performance indicator in standard operating procedure data.

5 . The method of claim 1 , wherein the operational specification data comprises standard operating procedure (SOP) data and pre-defined heuristic associated with the operation to be performed by at least one of a machine or an operator in the material handling environment.

6 . The method of claim 1 , wherein the data stream is configured to be used to perform at least one of:

monitoring movement of a plurality of containers in a replenishment zone in the material handling environment to one or more shuttle totes configured to store goods in an Automated Storage and Retrieval System (ASRS);

monitoring movement of pallets configured to stack a plurality of items associated with a plurality of characteristics;

monitoring movement of at least one of a robotic tool of a robotic manipulator or an operator in the material handling environment while executing an item picking operation;

identifying empty containers in the material handling environment;

monitoring movement of the empty containers in the material handling environment;

monitoring replenishment of the plurality of containers based on characteristics of the plurality of containers; or

monitoring movement of the plurality of containers placed on a gravity flow rack in the replenishment zone.

7 . The method of claim 1 further comprising:

scanning a container identifier associated with at least one container of a plurality of containers;

determining an occupied volume of the at least one container based on the data stream;

determining a current capacity of the at least one container based on a maximum capacity of the at least one container and the occupied volume of the at least one container; and

controlling item picking operation based on the determined current capacity of the at least one container.

8 . The method of claim 1 , further comprising:

generating an ordered sequence of container identifiers associated with a plurality of containers configured to be handled in the material handling environment; and

transmitting the ordered sequence of the container identifiers to a display device of a human-machine interface provided in the material handling environment.

9 . The method of claim 1 , further comprising:

displaying an ordered sequence of indicators of pick-to-lights of a pick-to-light system at locations of a plurality of containers corresponding to an ordered sequence of container identifiers;

capturing a sequence of actuation of the indicators of the pick-to-lights of the pick-to-light system; and

monitoring the performance status associated with the efficiency of an execution of the operation using the pick-to-light system.

10 . A non-transitory computer readable medium that stores thereon computer-executable instructions that, in response to being executed by a processor, perform operations comprising:

generating a machine learning model by using a data stream representative of a three-dimensional (3D) scan of a target area and operational specification data associated with an operation to be performed in a material handling environment as inputs to train the machine learning model;

comparing, by using the machine learning model, a first set of data points derived from the data stream comprising a 3D point cloud of the material handling environment with a second set of data points derived from the operational specification data; and

determining, by using the machine learning model, a performance status associated with an execution of the operation, wherein the performance status is associated with an efficiency of the execution of the operation.

11 . The non-transitory computer readable medium of claim 10 , wherein the operational specification data comprises standard operating procedure (SOP) data and pre-defined heuristic associated with the operation to be performed by at least one of a machine or an operator in the material handling environment.

12 . The non-transitory computer readable medium of claim 10 , wherein the computer-executable instructions, in response to being executed by the processor, perform operations further comprising:

generating a first feedback indicative of a degree of deviation from a predefined efficiency level; and

generating a second feedback indicative of an action to be performed so that an execution of the action causes the execution of the operation to meet the predefined efficiency level.

13 . The non-transitory computer readable medium of claim 10 , wherein the computer-executable instructions, in response to being executed by the processor, perform operations based on the data stream further comprising at least one of:

monitoring movement of a plurality of containers in a replenishment zone in the material handling environment to one or more shuttle totes configured to store goods in an Automated Storage and Retrieval System (ASRS);

monitoring movement of pallets configured to stack a plurality of items associated with a plurality of characteristics;

monitoring movement of at least one of a robotic tool of a robotic manipulator and an operator in the material handling environment while executing an item picking operation;

identifying empty containers in the material handling environment;

monitoring movement of the empty containers in the material handling environment;

monitoring replenishment of the plurality of containers based on characteristics of the plurality of containers; or

monitoring movement of the plurality of containers placed on a gravity flow rack in the replenishment zone.

14 . The non-transitory computer readable medium of claim 10 , wherein the computer-executable instructions, in response to being executed by the processor, perform operations further comprising:

determining a new key performance indicator associated with an operation of at least one of a machine, an operator, or a workflow process to be performed in the material handling environment based on analyzing of the operational specification data and the data stream by the machine learning model; and

modifying the operational specification data by including the new key performance indicator in standard operating procedure data.

15 . The non-transitory computer readable medium of claim 10 , wherein the computer-executable instructions, in response to being executed by the processor, perform operations further comprising:

scanning a container identifier associated with at least one container of a plurality of containers;

determining an occupied volume of the at least one container based on the data stream;

determining a current capacity of the at least one container based on a maximum capacity of the at least one container and the occupied volume of the at least one container; and

controlling item picking operation based on the determined current capacity of the at least one container.

16 . A material handling system comprising:

a computing system comprising a processor configured to:

obtain a data stream comprising a three-dimensional (3D) point cloud representative of a 3D-scan of a target area from a vision system;

obtain, from a warehouse execution system, operational specification data associated with an operation to be performed in a material handling environment;

generate a machine learning model by using the data stream and the operational specification data as inputs to train the machine learning model;

compare, by using the machine learning model, a first set of data points derived from the data stream comprising a 3D point cloud of the material handling environment with a second set of data points derived from the operational specification data; and

determine, by using the machine learning model, a performance status associated with an efficiency of an execution of the operation.

17 . The material handling system of claim 16 , wherein the operational specification data comprises a standard operating procedure (SOP) data and a pre-defined heuristic associated with the operation to be performed by at least one of a machine or an operator in the material handling environment.

18 . The material handling system of claim 16 , wherein the processor is configured to:

scan a container identifier associated with at least one container of a plurality of containers;

determine an occupied volume of the at least one container based on the data stream;

determine a current capacity of the at least one container based on a maximum capacity of the at least one container and the occupied volume of the at least one container; and

control item picking operation based on the determined current capacity of the at least one container.

19 . The material handling system of claim 16 , wherein the processor is configured to:

generate an ordered sequence of container identifiers associated with a plurality of containers configured to be handled in the material handling environment; and

transmit the ordered sequence of the container identifiers to a display device of a human-machine interface coupled to the processor.

20 . The material handling system of claim 16 , further comprising:

display an ordered sequence of indicators of pick-to-lights of a pick-to-light system at locations of a plurality of containers corresponding to an ordered sequence of container identifiers;

capture a sequence of actuation of the indicators of the pick-to-lights of the pick-to-light system; and

monitor the performance status associated with the efficiency of the execution of the operation using the pick-to-light system.

Assignments (1)

Continuity (2)

References Cited (36)