Granted Patent B2

US 12,327,345 · App. 17/874,135 · Granted Jun 10, 2025

Inventors: Sibo Zhang (Sunnyvale, CA); Liangjun Zhang (Cupertino, CA)

Assignee: Baidu USA LLC

G06T7/0004G06T7/11G06T7/73G06V10/225G06V10/764G06V10/82G06V20/52G08B21/02G06T2207/20021G06T2207/20084G06T2207/20132G06T2207/30108

Abstract

Claims (83)

1. A system for analyzing a working area, the system comprising:

one or more cameras that capture images of the working area;

one or more processors; and

a non-transitory computer-readable medium or media comprising one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:

segmenting the working area into one or more defined sub-areas using a working area segmentation neural network subsystem that receives image data from at least one of the one or more cameras and segments the working area into the one or more defined sub-areas;

detecting one or more objects in the working area using image data from at least one of the one or more cameras and an object detection neural network subsystem that receives the image data to produce, for each detected object from a set of one or more detected objects in the image data, a classification of the detected object and a bounding region data for the detected object; and

using a safety monitor subsystem to detect a safety issue, in which the safety monitor subsystem:

receives the one or more defined sub-areas of the working area from the working area segmentation neural network subsystem and, for each detected object from the set of one or more detected objects, its bounding region data from the object detection neural network subsystem;

determines, based upon one or more models, the one or more defined sub-areas of the working area, and the bounding region data from the object detection neural network subsystem, whether a safety issue exists; and

responsive to a safety issue existing, causes an alert to be made;

wherein the safety monitor subsystem determines a safety issue by:

using a bounding region data of a first detected object, a bounding region data of a second detected object, and at least one of the one or more defined sub-areas to monitor for safety by determining if a first threshold portion of the bounding region data of the first detected object and a second threshold portion of the bounding region data of the second detected object are within a same defined sub-area.

2. The system of claim 1 wherein at least one of the detected objects is equipment and the non-transitory computer-readable medium or media further comprises one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:

for the detected equipment, identifying a set of one or more action states of the equipment using an action recognition subsystem that, based upon image data from at least one of the one or more cameras, uses one or more models to identify the set of one or more action states of the equipment during a duration of the image data.

3. The system of claim 2 wherein the safety monitor subsystem is further configured to receive the set of one or more action states and uses one or more models to detect a safety issue by detecting a non-normal action state or a non-normal sequence of action states.

4. The system of claim 2 wherein the non-transitory computer-readable medium or media further comprises one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:

using the bounding region data of the detected equipment and the image data to generate a set of cropped images of the detected equipment; and

inputting the set of cropped images of the detected equipment into an action neural network model that identifies one or more action states of the detected equipment across the set of cropped images.

5. The system of claim 2 wherein at least one of the one or more models of the action recognition subsystem comprises:

a set of rules that identifies one or more action states of the detected equipment using a set of keypoints of the detected equipment across a set of images from the image data, in which the set of keypoints are obtained from a pose estimation subsystem that uses the bounding region data of the detected equipment and the image data from the object detection neural network subsystem to identify keypoints of the detected equipment across the set of images.

6. The system of claim 1 wherein the non-transitory computer-readable medium or media further comprises one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed:

for a detected object from the one or more detected objects, identifying a set of one or more action states of the detected object using an action recognition subsystem that, based upon image data from at least one of the one or more cameras, uses one or more models to identify the set of one or more action states of the detected object during a duration of the image data; and

using a productivity analysis subsystem that receives, for the detected object, the one or more of the action states from the action recognition subsystem and determines productivity for the detected object based upon a set of parameters comprising one or more object-related parameters.

7. The system of claim 1 wherein the safety monitor subsystem further determines a safety issue by performing at least one of the steps comprising:

using the bounding region data of the first detected object and the bounding region data of the second detected object to monitor for safety by determining if the bounding region data of the first detected object is within a threshold of the bounding region data of the second detected object; or

using the bounding region data of the first detected object and at least one of the one or more defined sub-areas to monitor for safety by determining if a threshold portion of the bounding region data of the first detected object is within one of the defined sub-areas.

8. The system of claim 1 wherein the non-transitory computer-readable medium or media further comprises one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed:

for a detected object from the one or more detected objects, refining the bounding region data using a set of keypoints for the detected object, in which the set of keypoints for the detected object is obtained from a pose estimation subsystem that uses the bounding region data of the detected object and image data from the object detection neural network subsystem to identify the set of keypoints.

9. The system of claim 1 wherein the non-transitory computer-readable medium or media further comprises one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed:

for a detected object from the one or more detected objects:

obtaining a set of keypoints for the detected object from a pose estimation neural network subsystem that uses the bounding region data of the detected object and image data from the object detection neural network subsystem to identify the set of keypoints; and

using at least some of the set of keypoints for the detected object to determine a safety issue responsive to detecting a non-normal keypoint orientation.

10. A computer-implemented method for analyzing a working area, the method comprising:

segmenting the working area into one or more defined sub-areas using a working area segmentation neural network subsystem that receives image data from at least one camera and segments the working area into the one or more defined sub-areas;

detecting one or more objects in the working area using image data from the at least one camera and an object detection neural network subsystem that receives the image data to produce, for each detected object from a set of one or more detected objects in the image data, a classification of the detected object and a bounding region data for the detected object; and

using a safety monitor subsystem to detect a safety issue, in which the safety monitor subsystem:

receives the one or more defined sub-areas of the working area from the working area segmentation neural network subsystem and, for each detected object from the set of one or more detected objects, its bounding region data from the object detection neural network subsystem;

determines, based upon one or more models, the one or more defined sub-areas of the working area, and the bounding region data from the object detection neural network subsystem, whether a safety issue exists; and

responsive to a safety issue existing, causes an alert to be made;

wherein the safety monitor subsystem determines a safety issue by:

using a bounding region data of a first detected object, a bounding region data of a second detected object, and at least one of the one or more defined sub-areas to monitor for safety by determining if a first threshold portion of the bounding region data of the first detected object and a second threshold portion of the bounding region data of the second detected object are within a same defined sub-area.

11. The computer-implemented method of claim 10 further comprises:

for a detected object from the set of one or more detected objects, defining a set of one or more action states of the detected object using an action recognition subsystem that, based upon image data from the at least one camera, uses one or more models to define the set of one or more action states of the detected object during a duration of the image data.

12. The computer-implemented method of claim 11 wherein:

the action recognition subsystem obtains the set of one or more action states of the detected object by performing steps comprising:

using the bounding region data of the detected object and the image data to generate a set of cropped images of the detected object; and

inputting the set of cropped images of the detected object into an action neural network model that identifies the set of one or more action states of the detected object across the set of cropped images; and

the computer-implemented method further comprises:

using at least one or more of the action states to detect a safety issue responsive to detecting a non-normal action state or a non-normal sequence of action states.

13. The computer-implemented method of claim 11 wherein at least one of the one or more models of the action recognition subsystem comprises:

a set of rules that identifies one or more action states of the detected object using a set of keypoints of the detected object across a set of images from the image data, in which the set of keypoints are obtained from a pose estimation subsystem that uses the bounding region data of the detected object, and the image data from the object detection neural network subsystem to identify keypoints of the detected object across the set of images.

14. The computer-implemented method of claim 10 wherein the safety monitor subsystem further determines a safety issue by performing at least one of the steps comprising:

using the bounding region data of the first detected object and the bounding region data of the second detected object to monitor for safety by determining if the bounding region data of the first detected object is within a threshold of the bounding region data of the second detected object; or

using the bounding region data of the first detected object and at least one of the one or more defined sub-areas to monitor for safety by determining if a threshold portion of the bounding region data of the first detected object is within one of the defined sub-areas.

15. The computer-implemented method of claim 10 wherein the step of producing a bounding region data for the detected object further comprises:

for a detected object from the one or more detected objects, refining an initial bounding region data of the detected object to obtain the bounding region data by performing steps comprising:

obtaining a set of keypoints for the detected object using a pose estimation subsystem that uses the initial bounding region data of the detected object and image data from the object detection neural network subsystem to identify the set of keypoints of the detected object; and

using the set of keypoints for the detected object and the pose estimation subsystem to determine a pose for the detected object; and

using the pose to refine the initial bounding region data for the detected object.

16. The computer-implemented method of claim 10 further comprising:

for a detected object from the one or more detected objects:

obtaining a set of keypoints for the detected object from a pose estimation neural network subsystem that uses the bounding region data of the detected object and image data from the object detection neural network subsystem to identify the set of keypoints; and

using at least some of the set of keypoints for the detected object to determine a safety issue responsive to detecting a non-normal keypoint orientation.

17. A system for analyzing a working area, the system comprising:

one or more processors;

a non-transitory computer-readable medium or media comprising one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:

segmenting the working area into one or more defined sub-areas using a working area segmentation neural network subsystem that receives image data from at least one camera and segments the working area into the one or more defined sub-areas;

detecting one or more objects in the working area using image data from the at least one camera and an object detection neural network subsystem that receives the image data to produce, for each detected object from a set of one or more detected objects in the image data, a classification of the detected object and a bounding region data for the detected object; and

using a safety monitor subsystem to detect a safety issue, in which the safety monitor subsystem:

receives the one or more defined sub-areas of the working area from the working area segmentation neural network subsystem and, for each detected object from the set of one or more detected objects, its bounding region data from the object detection neural network subsystem;

determines, based upon one or more models, the one or more defined sub-areas of the working area, and the bounding region data from the object detection neural network subsystem, whether a safety issue exists; and

responsive to a safety issue existing, causes an alert to be made;

wherein the safety monitor subsystem determines a safety issue by:

using a bounding region data of a first detected object, a bounding region data of a second detected object, and at least one of the one or more defined sub-areas to monitor for safety by determining if a first threshold portion of the bounding region data of the first detected object and a second threshold portion of the bounding region data of the second detected object are within a same defined sub-area.

18. The system of claim 17 wherein the non-transitory computer-readable medium or media further comprises one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:

for a detected object from the set of one or more detected objects, defining a set of one or more action states of the detected object using an action recognition subsystem that, based upon image data from the at least one camera, uses one or more models to define the set of one or more action states of the detected object during a duration of the image data.

19. The system of claim 18 wherein:

the action recognition subsystem obtains the set of one or more action states of the detected object using either a rules-based model, a neural network-based model, or both; and

the non-transitory computer-readable medium or media further comprises one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:

using at least one or more of the action states to detect a safety issue responsive to detecting a non-normal action state or a non-normal sequence of action states.

20. The system of claim 17 wherein the safety monitor subsystem further determines a safety issue by performing at least one of the steps comprising:

using the bounding region data of the first detected object and the bounding region data of the second detected object to monitor for safety by determining if the bounding region data of the first detected object is within a threshold of the bounding region data of the second detected object; or

using the bounding region data of the first detected object and at least one of the one or more defined sub-areas to monitor for safety by determining if a threshold portion of the bounding region data of the first detected object is within one of the defined sub-areas.

Assignments (1)

Continuity (2)

References Cited (38)