IP Library Granted Patent US 11,790,575
Granted Patent B2
US 11,790,575 · App. 17/812,830 · Granted Oct 17, 2023

Object detection training based on artificially generated images

Inventors: Ian Cinnamon (Sherman Oaks, CA); Bruno Brasil Ferrari Faviero (San Francisco, CA); Simanta Gautam (Charlottesville, VA)
Assignee: Rapiscan Laboratories, Inc.
G06T11/003G01V5/00G06T7/12G06T9/002G06T15/503G06T17/30
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,790,575
App. No.
17/812,830
Granted
Oct 17, 2023
Kind
B2
Abstract

Technology disclosed herein may involve a computing system that (i) based on an image of a target object of a given class of object and at least one GAN configured to generate artificial images of the given class of object, generates an artificial image of the target object that is substantially similar to real-world images of objects of the given class of objects captured by real-world scanning devices, (ii) based on an image of a receptacle, selects an insertion location within the receptacle in the image of the receptacle to insert the artificial image of the target object, (iii) generates a combined image of the receptacle and the target object, wherein generating the combined image comprises inserting the artificial image of the target object into the image of the receptacle at the insertion location, and (iv) trains one or more object detection algorithms with the combined image of the receptacle and the target object.

Claims (39)

1. A computing system comprising:

at least one processor configured to receive data associated with an input image comprising a target object, wherein the target object is one of a plurality of classes of objects;

non-transitory computer-readable medium;

program instructions that are stored on the non-transitory computer-readable medium that are executable by the at least one processor such that the computing system is configured to:

based upon said data, generate a plurality of feature maps of the input image;

based upon said data, select at least one feature map of the plurality of feature maps, wherein the at least one feature map is selected based on whether it is most likely to generate an accurate simulated image of the target object relative to other ones of the plurality of feature maps;

based on the input image and the selected at least one feature map, generate an artificial image of the target object that is substantially similar to real-world images of objects of the one of the plurality of classes of objects captured by real-world scanning devices;

based on an image of a receptacle, select an insertion location within the receptacle in the image of the receptacle to insert the artificial image of the target object;

generate a combined image of the receptacle and the target object, wherein generating the combined image comprises inserting the artificial image of the target object into the image of the receptacle at the insertion location;

apply a variation to the combined image, wherein the variation comprises a change in at least one of an intensity, an obscuration, a noise, a magnification, a rotation, or a Z-effective encoding of the combined image; and

train one or more object detection algorithms with the combined image of the receptacle and the target object.

2. The computing system of claim 1 , comprising program instructions that are stored on the non-transitory computer-readable medium and are executable by the at least one processor to use a set of neighboring input values to generate a representation of features identified from the input image and thereby generate each of the plurality of feature maps.

3. The computing system of claim 2 , comprising program instructions that are stored on the non-transitory computer-readable medium and are executable by the at least one processor to output the plurality of features to at least one encoder to generate the plurality of feature maps.

4. The computing system of claim 3 , comprising program instructions that are stored on the non-transitory computer-readable medium and are executable by the at least one processor to receive, in a decoder, the generated plurality of feature maps.

5. The computing system of claim 1 , comprising program instructions that are stored on the non-transitory computer-readable medium and are executable by the at least one processor to select at least one feature map from the plurality of feature maps using a neural network.

6. The computing system of claim 5 , wherein the neural network is at least one of a generator neural network and a discriminator neural network, and wherein the selected at least one feature map is configured to generate artificial images of the one of the plurality of classes of objects based on the generator neural network and/or the discriminator neural network having been trained such that an error function in said generator neural network and/or said discriminator neural network is minimized.

7. The computing system of claim 1 , wherein the receptacle comprises a baggage item, and wherein the image of the receptacle comprises a real-world image of the baggage item captured by a real-world scanning device.

8. The computing system of claim 1 , wherein the program instructions that are executable by the at least one processor to select the at least one feature map from the plurality of feature maps comprise instructions that, when executed by the at least one processor, makes a determination that the target object belongs to the one of a plurality of classes of objects based on the image of the target object, and wherein the at least one feature map that is selected from the plurality of feature maps is configured to generate artificial images of the one of a plurality of classes of objects based on the determination.

9. The computing system of claim 1 , wherein the program instructions that are executable by the at least one processor further comprise instructions that, when executed by the at least one processor, overlays the artificial image of the target object within the image of the receptacle with at least a portion of a background object from the image of the receptacle.

10. The computing system of claim 1 , wherein the input image is a projection image and the data is metadata.

11. The computing system of claim 1 , wherein the instructions that are executable by the at least one processor further comprise instructions that, when executed, establish boundaries for the variation and randomly sample values within the boundaries of the variation.

12. A computer-implemented method comprising:

receiving data associated with an input image, wherein the input image comprises a target object of a given class from a plurality of classes of objects;

generating a plurality of feature maps;

selecting at least one feature map from the plurality of feature maps based on said data; wherein the at least one feature map is selected based on whether it is most likely to generate an accurate simulated image of the target object relative to other ones of the plurality of the feature maps;

based on an image of a target object of a given class of object and the selected at least one feature map, generating an artificial image of the target object that is substantially similar to real-world images of objects of the given class of objects captured by real-world scanning devices;

based on an image of a receptacle, selecting an insertion location within the image of the receptacle to insert the generated artificial image of the target object;

generating a combined image of the receptacle and the generated artificial image of the target object, wherein generating the combined image comprises inserting the artificial image of the target object into the image of the receptacle at the insertion location;

applying a variation to the combined image, wherein the variation comprises a change in at least one of an intensity, an obscuration, a noise, a magnification, a rotation, or a Z-effective encoding of the combined image; and

training one or more object detection algorithms with the combined image of the receptacle and the target object.

13. The computer-implemented method of claim 12 , wherein generating the plurality of feature maps comprises generating a set of neighboring input values to generate a representation of features identified from said input image.

14. The computer-implemented method of claim 13 , further comprising transmitting the generated plurality of feature maps to at least one encoder and receiving outputs from the at least one encoder in at least one decoder.

15. The computer-implemented method of claim 12 , wherein selecting the at least one feature map from the plurality of feature maps comprises:

based on the image of the target object, making a determination that the target object belongs to the given class of objects and based on the determination, selecting from the plurality of feature maps the at least one feature map that is configured to generate artificial images of the given class of object from the plurality of classes of objects.

16. The computer-implemented method of claim 12 , further comprising overlaying the artificial image of the target object within the image of the receptacle with at least a portion of a background object from the image of the receptacle.

17. The computer-implemented method of claim 12 , further comprising establishing boundaries for the variation and randomly sampling values within the boundaries of the variation.

18. The computer-implemented method of claim 12 , wherein the input image is a projection image and the data is metadata.

19. The computer-implemented method of claim 12 , further comprising selecting the at least one feature map from the plurality of feature maps using a neural network.

20. The computer-implemented method of claim 19 , wherein the neural network is a generator and/or discriminator neural network.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 15, 2022
From: CINNAMON, IAN; FAVIERO, BRUNO BRASIL FERRARI; GAUTAM, SIMANTA
To: SYNAPSE TECHNOLOGY CORPORATION
Reel/Frame 060521/0703 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 15, 2022
From: SYNAPSE TECHNOLOGY CORPORATION
To: RAPISCAN LABORATORIES, INC.
Reel/Frame 060521/0749 →
Continuity (5)
Continuation 16658513 · Oct 21, 2019
Continuation 15799274 · Oct 31, 2017
Continuation 15727108 · Oct 6, 2017
Provisional Application 62547626 · Aug 18, 2017
Related Publication 20230010382A1 · Jan 12, 2023
Cited By (1)
US 12,385,854