IP Library › Granted Patent US 12,205,359
Granted Patent B2
US 12,205,359 · App. 17/456,667 · Granted Jan 21, 2025

Two-stage screening technique for prohibited objects at security checkpoints using image segmentation

Inventors: Mohamed Nooman Ahmed (Leesburg, VA); Fayin Li (Herndon, VA)
Assignee: International Business Machines Corporation
G06V10/82G01N23/046G01N23/083G01N23/10G06N3/08G06Q10/08G06T7/10G06T7/70G06V10/764G06V20/52G01N2223/04G01N2223/419G06T2200/04G06T2207/10081G06T2207/20081G06T2207/20084G06T2207/30232G06V2201/05
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,205,359
App. No.
17/456,667
Granted
Jan 21, 2025
Kind
B2
Abstract

A system and method for classifying compartments at a security checkpoint includes classifying a compartment into a first category or a second category using a first stage neural network that analyzes a three-dimensional representation of the compartment extracted from an imaging device coupled to the computing system, and in response to classifying the compartment into the second category, screening the compartment using a second stage neural network that performs image segmentation to isolate a hazardous object present in the compartment.

Claims (31)

1. A method comprising:

classifying, by a processor of a computing system, a compartment into a first category or a second category using a first stage neural network that analyzes a three-dimensional representation of the compartment extracted from an imaging device coupled to the computing system, wherein the classifying includes inputting a single image into the first stage neural network to classify the compartment, the single image representing a volume of the compartment using Maximum Intensity Orthogonal 2D Planes (MIOP) extracted from scanned images output by the imaging device, which are stacked into the single image; and

in response to classifying the compartment into the second category, screening, by the processor, the compartment using a second stage neural network that performs image segmentation to isolate a hazardous object present in the compartment;

wherein the first stage neural network classifies the compartment into the first category or the second category regardless of a shape or a material of an object contained within the compartment, without requiring human inspection of an output of the imaging device.

2. The method of claim 1 , wherein the first category refers to the compartment being classified as no threat, and the compartment is automatically cleared at a security checkpoint.

3. The method of claim 1 , wherein the first stage neural network classifies the compartment as containing potentially hazardous objects, and the Maximum Intensity Orthogonal 2D Planes (MIOP) are extracted from Computed Tomography (CT) scan images.

4. The method of claim 3 , wherein the first stage neural network is a trained two-dimensional convolutional neural network (CNN).

5. The method of claim 1 , wherein the screening includes inputting a single image into the second stage neural network configured to capture spatial relationships in three-dimensional space from the single image, further wherein the single image is an output of the first stage neural network.

6. The method of claim 5 , wherein the second stage neural network is a regional proposal network (RPN) based on UNET architecture comprising a first path and a second path capturing context of the single image, the first path being an encoder configured as a stack of convolutional layers and max pooling layers and the second path is a decoder configured as a symmetric expanding path enabling localization using transposed convolutions extracted from the imaging device.

7. The method of claim 1 , wherein, as a function of leveraging the second stage neural network, a location of the hazardous object within the compartment is isolated and displayed on a display device.

8. A computing system, comprising:

a processor;

a memory device coupled to the processor; and

a computer readable storage device coupled to the processor, wherein the storage device contains program code executable by the processor via the memory device to implement a method comprising:

classifying, by the processor, a compartment into a first category or a second category using a first stage neural network that analyzes a three-dimensional representation of the compartment extracted from an imaging device coupled to the computing system, wherein the classifying includes inputting a single image into the first stage neural network to classify the compartment, the single image representing a volume of the compartment using Maximum Intensity Orthogonal 2D Planes (MIOP) extracted from scanned images output by the imaging device, which are stacked into the single image; and

in response to classifying the compartment into the second category, screening, by the processor, the compartment using a second stage neural network that performs image segmentation to isolate a hazardous object present in the compartment;

wherein the first stage neural network classifies the compartment into the first category or the second category regardless of a shape or a material of an object contained within the compartment, without requiring human inspection of an output of the imaging device.

9. The computing system of claim 8 , wherein the first category refers to the compartment being classified as no threat, and the compartment is automatically cleared at a security checkpoint.

10. The computing system of claim 8 , wherein the first stage neural network classifies the compartment as containing potentially hazardous objects, and the Maximum Intensity Orthogonal 2D Planes (MIOP) are extracted from Computed Tomography (CT) scan images.

11. The computing system of claim 10 , wherein the first stage neural network is a trained two-dimensional convolutional neural network (CNN).

12. The computing system of claim 8 , wherein the screening includes inputting a single image into the second stage neural network configured to capture spatial relationships in three-dimensional space from the single image, further wherein the single image is an output of the first stage neural network.

13. The computing system of claim 12 , wherein the second stage neural network is a regional proposal network (RPN) based on UNET architecture comprising a first path and a second path capturing context of the single image, the first path being an encoder configured as a stack of convolutional layers and max pooling layers and the second path is a decoder configured as a symmetric expanding path enabling localization using transposed convolutions extracted from the imaging device.

14. The computing system of claim 8 , wherein, as a function of leveraging the second stage neural network, a location of the hazardous object within the compartment is isolated and displayed on a display device.

15. A computer program product, comprising a computer readable hardware storage device storing a computer readable program code, the computer readable program code comprising an algorithm that when executed by a computer processor of a computing system implements a method comprising:

classifying, by the processor, a compartment into a first category or a second category using a first stage neural network that analyzes a three-dimensional representation of the compartment extracted from an imaging device coupled to the computing system, wherein the classifying includes inputting a single image into the first stage neural network to classify the compartment, the single image representing a volume of the compartment using Maximum Intensity Orthogonal 2D Planes (MIOP) extracted from scanned images output by the imaging device, which are stacked into the single image; and

in response to classifying the compartment into the second category, screening, by the processor, the compartment using a second stage neural network that performs image segmentation to isolate a hazardous object present in the compartment;

wherein the first stage neural network classifies the compartment into the first category or the second category regardless of a shape or a material of an object contained within the compartment, without requiring human inspection of an output of the imaging device.

16. The computer program product of claim 15 , wherein the first category refers to the compartment being classified as no threat, and the compartment is automatically cleared at a security checkpoint.

17. The computer program product of claim 15 , wherein the the first stage neural network classifies the compartment as containing potentially hazardous objects, and the Maximum Intensity Orthogonal 2D Planes (MIOP) are extracted from Computed Tomography (CT) scan images.

18. The computer program product of claim 17 , wherein the screening includes inputting a single image into the second stage neural network configured to capture spatial relationships in three-dimensional space from the single image, further wherein the single image is an output of the first stage neural network.

19. The computer program product of claim 18 , wherein the second stage neural network is a regional proposal network (RPN) based on UNET architecture comprising a first path and a second path capturing context of the single image, the first path being an encoder configured as a stack of convolutional layers and max pooling layers and the second path is a decoder configured as a symmetric expanding path enabling localization using transposed convolutions extracted from the imaging device.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 29, 2021
From: AHMED, MOHAMED NOOMAN; LI, FAYIN
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 058224/0898 →
Continuity (1)
Related Publication 20230169619A1 · Jun 1, 2023
References Cited (71)
US 6026171A · Hiraoglu · 2000 [cited by applicant]
US 9268058B2 · Peschmann · 2016 [cited by applicant]
US 9996890B1 · Cinnamon · 2018 [cited by applicant]
US 10013750B2 · Zhang · 2018 [cited by applicant]
US 10014076B1 · David · 2018 [cited by applicant]
US 10366293B1 · Faviero · 2019 [cited by examiner]
US 11093803B2 · Ahmed · 2021 [cited by applicant]
US 11106930B2 · Ahmed · 2021 [cited by applicant]
US 20090063118A1 · Dachille · 2009 [cited by examiner]
US 20090175411A1 · Gudmundson · 2009 [cited by examiner]
US 20170242148A1 · Yu · 2017 [cited by applicant]
US 20170365118A1 · Nurbegovic · 2017 [cited by applicant]
US 20190209116A1 · Sjöstrand · 2019 [cited by applicant]
US 20190244346A1 · Schafer · 2019 [cited by applicant]
US 20200103548A1 · Yu · 2020 [cited by applicant]
US 20200167943A1 · Kim · 2020 [cited by examiner]
US 20200193666A1 · Cinnamon · 2020 [cited by applicant]
US 20200320685A1 · Anssari Moin · 2020 [cited by applicant]
US 20200394417A1 · Ahmed · 2020 [cited by applicant]
US 20200394430A1 · Ahmed · 2020 [cited by applicant]
US 20200394442A1 · Ahmed · 2020 [cited by applicant]
CN 107607562B · 2020 [cited by applicant]
KR 102063859B1 · 2020 [cited by applicant]
Zanuttigh et al., Deep Learning for 3D Shape Classification From Multiple Depth Maps, Department of Information Engineering, University of Padova, Italy, 2017, https://ieeexplore.IEEE.org/document/8296956, 5 pages. [cited by applicant]
Bu et al., 3D shape recognition and retrieval based on multi-modality deep learning, vol. 259, Oct. 11, 2017, Retrieved from Internet on Jan. 5, 2022: https://www.sciencedirect.com/science/article/abs/pii/S0925231217302… [cited by applicant]
Le et al., PointGrid: a Deep Network for 3D Shape Understanding, University of Missouri—Columbia, 2018, https://openaccess.thecvf.com/content_cvpr_2018/papers/Le_PointGrid_A_Deep_CVPR_2018_paper.pdf, 11 pages. [cited by applicant]
Wu et al., 3D ShapeNets: a Deep Representation for Volumetric Shapes, Cornell University, https://arxiv.org/abs/1406.5670, Submitted on Jun. 22, 2014, last revised Apr. 15, 2015, 9 pages. [cited by applicant]
Qi et al., PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation, Cornell University, https://arxiv.org/abs/1612.00593, Submitted on Dec. 2, 2016, last revised Apr. 10, 2017, 9 pages. [cited by applicant]
Kalogerakis et al., 3D Shape Segmentation with Projective Convolutional Networks, Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR) 2017, https://people.cs.umass.edu/˜kalo/papers/shapepfcn/ShapePFCN… [cited by applicant]
Cao et al., Learning to Reconstruct High-quality 3D Shapes with Cascaded Fully Convolutional Networks, https://openaccess.thecvf.com/content_ECCV_2018/papers/Yan-Pei_Cao_Learning_to_Reconstruct_ECCV_2018_paper.pdf. 18 p… [cited by applicant]
Lippert et al., Exploring DeepMedic for the purpose of segmenting white matter hyperintensity lesions, Feb. 27, 2018, https://www.spiedigitallibrary.org/conference-proceedings-of-spie/10575/105752F/Exploring-DeepMedic-f… [cited by applicant]
Minto et al., Deep Learning for 3D Shape Classification Based on Volumetric Density and Surface Approximation Clues, Published Sep. 1, 2017 Computer Science 2017 IEEE International Conference on Image Processing, https:… [cited by applicant]
Kao et al., Brain Tumor Segmentation and Tractographic Feature Extraction from Structural MR Images for Overall Survival Prediction, https://arxiv.org/pdf/1807.07716.pdf, arXiv:1807.07716v3 [cs.CV] Nov. 12, 2018, 14 pag… [cited by applicant]
Oktay et al., Attention U-Net: Learning Where to Look for the Pancreas, https://arxiv.org/pdf/1804.03999.pdf, arXiv:1804.03999v3 [cs.CV] May 20, 2018, 10 pages. [cited by applicant]
Cicek et al., 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation, https://arxiv.org/abs/1606.06650.pdf, arXiv:1606.06650v1 [cs.CV] Jun. 21, 2016, 8 pages. [cited by applicant]
Roth et al., Deep learning and its application to medical image segmentation, https://arxiv.org/pdf/1803.08691.pdf, arXiv:1803.08691v1 [cs.CV] Mar. 23, 2018, 6 pages. [cited by applicant]
Heimann et al., Statistical shape models for 3D medical image segmentation: a review, Medical Image Analysis vol. 13, Issue 4, Aug. 2009, pp. 543-563, https://www.sciencedirect.com/science/article/abs/pii/S1361841509000… [cited by applicant]
Socher et al., Convolutional-Recursive Deep Learning for 3D Object Classification, https://proceedings.neurips.cc/paper/2012/file/3eae62bba9ddf64f69d49dc48e2dd214-Paper.pdf, Computer Science Department, Stanford Univers… [cited by applicant]
Isin, et al., Review of MRI-based Brain Tumor Image Segmentation Using Deep Learning Methods, Procedia Computer Science vol. 102, 2016, pp. 317-324, https://www.sciencedirect.com/science/article/pii/S187705091632587X, 8… [cited by applicant]
Bui et al., 3D Densely Convolutional Networks for Volumetric Segmentation, arXiv: 1709.03199v2 [cs.CV] Sep. 13, 2017, https://arxiv.org/pdf/1709.03199.pdf, 7 pages. [cited by applicant]
Bernal et al., Quantitative analysis of patch-based fully convolutional neural networks for tissue segmentation on brain magnetic resonance imaging, inarXiv: 1801.06457v2 [cs.CV] Feb. 19, 2018, https://arxiv.org/pdf/180… [cited by applicant]
Suk et al., Deep ensemble learning of sparse regression models for brain disease diagnosis, Medical Image Analysis vol. 37, Apr. 2017, pp. 101-113, Retrieved from Internet on Jan. 11, 2022, https://www.sciencedirect.com… [cited by applicant]
Szegedy et al., Going deeper with convolutions, arXiv: 1409.4842v1 [cs.CV] Sep. 17, 2014, https://arxiv.org/pdf/1409.4842.pdf, 12 pages. [cited by applicant]
Sarkar et al., Learning 3D Shapes as Multi-Layered Height-maps using 2D Convolutional Networks, arXiv:1807.08485v2 [cs.CV] Jul. 26, 2018, https://arxiv.org/pdf/1807.08485.pdf, 23 pages. [cited by applicant]
Roth et al., An application of cascaded 3D fully convolutional networks for medical image segmentation, arXiv:1803.05431v2 [cs.CV] Mar. 20, 2018, https://arxiv.org/pdf/1803.05431.pdf, 21 pages. [cited by applicant]
Litjens, et al., A Survey on Deep Learning in Medical Image Analysis, arXiv:1702.05747v2 [cs.CV] Jun. 4, 2017, https://arxiv.org/pdf/1702.05747.pdf, 38 pages. [cited by applicant]
Krizhevsky et al., ImageNet Classification with Deep Convolutional Neural Networks, https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf, 9 pages. [cited by applicant]
Ronneberger et al., U-Net: Convolutional Networks for Biomedical Image Segmentation, arXiv:1505.04597v1 [cs. CV] May 18, 2015, https://arxiv.org/pdf/1505.04597.pdf, 8 pages. [cited by applicant]
He et al., Deep Residual Learning for Image Recognition, arXiv:1512.03385v1 [cs.CV] Dec. 10, 2015, https://arxiv.org/pdf/1512.03385.pdf, 12 pages. [cited by applicant]
Zheng et al., 3D Deep Learning for Efficient and Robust Landmark Detection in Volumetric Data, Conference paper First Online: Nov. 18, 2015, https://link.springer.com/chapter/10.1007/978-3-319-24553-9_69, 4 pages. [cited by applicant]
Bakator et al., Deep Learning and Medical Diagnosis: a Review of Literature, Received: Jun. 20, 2018; Accepted: Aug. 14, 2018; Published: Aug. 17, 2018, Multimodal Technologies and Interact. 2018, 2, 47; doi: 10.3390/mt… [cited by applicant]
Huang et al., Densely Connected Convolutional Networks, arXiv:1608.06993v5 [cs.CV] Jan. 28, 2018, https://arxiv.org/pdf/1608.06993.pdf, 9 pages. [cited by applicant]
Dolz et al., HyperDense-Net: a hyper-densly connected CNN for multi-modal image segmentations, arXiv:1804.02967v2 [cs.CV] Mar. 2, 2019, https://arxiv.org/abs/1804.02967v2, 13 pages. [cited by applicant]
Anomalous Object detection in mm-wave imagery, IP.com No. IPCOM000200693D, IP.com Electronic Publication Date: Oct. 25, 2010, copyright: HRL Laboratories, LLC, 6 pages. [cited by applicant]
Brock, Andrew, et al.; Generative and Discriminative Voxel Modeling with Convolutional Neural Networks. Aug. 16, 2016, arxiv.org/abs/1608.04236; 9 pages. [cited by applicant]
Collaborative Mobile Device Enabled In-Baggage Suspicious Object Detection, Authors: et. al.: Disclosed Anonymously, IP.com No. IPCOM000258674D, IP.com Electronic Publication Date: Jun. 3, 2019, 7 pages. [cited by applicant]
Flitton, G. et al., Object Classification in 3D Baggage Security Computed Tomography Imagery using Visual Codebooks, Jan. 2015, 26 pages. [cited by applicant]
Garcia-Garcia, Alberto, et al.; A Review on Deep Learning Techniques Applied to Semantic Segmentation; arxiv:1704.06857v1; Apr. 22, 2017; 23 pages. [cited by applicant]
Guo, Zhihui, et al. “Deep Logismos: Deep Learning Graph-Based 3D Segmentation of Pancreatic Tumors on CT Scans.” 25 2018, arxiv.org/abs/1801.08599; 4 pages. [cited by applicant]
He, Kaiming, et al.; “Mask R-CNN.” [Astro-Ph/0005112] a Determination of the Hubble Constant from Cepheid Distance a Model of the Local Peculiar Velocity Field, American Physical Society, Jan. 24, 2018, arxiv.org/abs/17… [cited by applicant]
Kamnitsas, Konstantinos et al.; “Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation.” 2017, arxiv.org/abs/1603.05959; 49 pages. [cited by applicant]
Kanezaki, Asako, et al. RotationNet: Joint Object Categorization and Pose Estimation Using Multiviews from Unsupervised Viewpoints. Mar. 23, 2018, arxiv.org/abs/1603.06208; 24 pages. [cited by applicant]
Mell, Peter et al.; “The NIST Definition of Cloud Computing;” National Institute of Standards and Technology; Special Publication 800-145; Sep. 2011; 7 pages. [cited by applicant]
Menegola, Afonso, et al. “Towards Automated Melanoma Screening: Exploring Transfer Learning Schemes.” American PhysicaSociety, Sep. 5, 2016, arxiv.org/abs/1609.01228; 4 pages. [cited by applicant]
Method and system for airport baggage security checking with holographic object navigation, Authors: et. al.: Disclosed Anonymously, IP.com No. IPCOM000246751D, IP.com Electronic Publication Date: Jun. 29, 2016, 3 pages. [cited by applicant]
Method and System for Interpreting Scanned Security Images using Cognitive Solutions, Authors: et. al.: Disclosed Anonymously, IP.com No. IPCOM000255971D, IP.com Electronic Publication Date: Oct. 26, 2018, 3 pages. [cited by applicant]
Mouton, A. et al., A review of automated image understanding within 3D baggage computed tomography security screening, Article in Journal of X-Ray Science and Technology, Sep. 2015, 50 pages. [cited by applicant]
Murphy, Jason A.; List of IBM Patents or Patent Applications Treated as Related; Nov. 24, 2021; 1 page. [cited by applicant]
Prasoon et al., Deep Feature Learning for Knee Cartilage Segmentation Using a Triplanar Convolutional Neural Network, MICCAI 2013, Part II, LNCS 8150, pp. 246-353. [cited by applicant]
Q. Wang et al., “Multi-Class 3D Object Detection Within Volumetric 3D Computed Tomography Baggage Security Screening Imagery,” [Submitted on Aug. 3, 2020], https://arxiv.org/abs/2008.01218. [cited by applicant]
Zelener, Allan; Survey of Object Classification in 3D Range Scans; 2017; 32 pages. [cited by applicant]