IP Library Granted Patent US 11,423,592
Granted Patent B2
US 11,423,592 · App. 16/658,513 · Granted Aug 23, 2022

Object detection training based on artificially generated images

Inventors: Ian Cinnamon (Sherman Oaks, CA); Bruno Brasil Ferrari Faviero (Palo Alto, CA); Simanta Gautam (Charlottesville, VA)
Assignee: Rapiscan Laboratories, Inc.
G06T11/003G01V5/00G06T7/12G06T9/002G06T15/503G06T17/30
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,423,592
App. No.
16/658,513
Granted
Aug 23, 2022
Kind
B2
Abstract

Technology disclosed herein may involve a computing system that (i) based on an image of a target object of a given class of object and at least one GAN configured to generate artificial images of the given class of object, generates an artificial image of the target object that is substantially similar to real-world images of objects of the given class of objects captured by real-world scanning devices, (ii) based on an image of a receptacle, selects an insertion location within the receptacle in the image of the receptacle to insert the artificial image of the target object, (iii) generates a combined image of the receptacle and the target object, wherein generating the combined image comprises inserting the artificial image of the target object into the image of the receptacle at the insertion location, and (iv) trains one or more object detection algorithms with the combined image of the receptacle and the target object.

Claims (43)

1. A computing system comprising:

at least one processor configured to receive data associated with an input image comprising a target object, wherein the target object is one of a plurality of classes of objects;

a plurality of neural networks;

a non-transitory computer-readable medium;

program instructions that are stored on the non-transitory computer-readable medium that are executable by the at least one processor such that the computing system is configured to:

based upon said data, select at least one neural network of the plurality of neural networks, wherein the at least one neural network is selected based on whether it is most likely to generate an accurate simulated image of the target object relative to other ones of the plurality of neural networks;

based on the input image and the selected at least one neural network, generate an artificial image of the target object that is substantially similar to real-world images of objects of the one of the plurality of classes of objects captured by real-world scanning devices;

based on an image of a receptacle, select an insertion location within the receptacle in the image of the receptacle to insert the artificial image of the target object;

generate a combined image of the receptacle and the target object, wherein generating the combined image comprises inserting the artificial image of the target object into the image of the receptacle at the insertion location;

apply a variation to the combined image, wherein the variation comprises a change in at least one of an intensity, an obscuration, a noise, a magnification, a rotation, or a Z-effective encoding of the combined image; and

train one or more object detection algorithms with the combined image of the receptacle and the target object.

2. The computing system of claim 1 , wherein the at least one neural network comprises a generator neural network and a discriminator neural network, and wherein the at least one neural network is configured to generate artificial images of the one of the plurality of classes of objects based on the generator neural network and the discriminator neural network having been trained such that an error function was minimized.

3. The computing system of claim 1 , wherein the receptacle comprises a baggage item, and wherein the image of the receptacle comprises a real-world image of the baggage item captured by a real-world scanning device.

4. The computing system of claim 1 , wherein the program instructions that are executable by the at least one processor such that the computing system is configured to select the at least one neural network from the plurality of neural networks comprise instructions that are executable by the at least one processor such that the computing system is configured to: based on the image of the target object, make a determination that the target object belongs to the one of a plurality of classes of objects, wherein the at least one neural network that is selected from the plurality of neural networks is configured to generate artificial images of the one of a plurality of classes of objects, based on the determination.

5. The computing system of claim 1 , wherein the program instructions that are executable by the at least one processor further comprise instructions that, when executed by the at least one processor overlays the artificial image of the target object within the image of the receptacle with at least a portion of a background object from the image of the receptacle.

6. A non-transitory computer-readable medium comprising program instructions stored thereon that are executable by at least one processor such that a computing system is configured to:

receive data associated with an input image comprising a target object of a given class from a plurality of classes of objects;

based on said data, select at least one neural network of a plurality of neural networks, wherein the at least one neural network is selected based on whether it is most likely to generate an accurate simulated image of the target object relative to other ones of the plurality of the neural networks;

based on the input image and the selected at least one neural network, generate an artificial image of the target object that is substantially similar to real-world images of objects of the given class of objects captured by real-world scanning devices;

based on an image of a receptacle, select an insertion location within the receptacle in the image of the receptacle to insert the artificial image of the target object;

generate a combined image of the receptacle and the target object, wherein generating the combined image comprises inserting the artificial image of the target object into the image of the receptacle at the insertion location;

apply a variation to the combined image, wherein the variation comprises a change in at least one of an intensity, an obstruction, a noise, a magnification, a rotation, or a Z-effective encoding of the combined image; and

train one or more object detection algorithms with the combined image of the receptacle and the target object.

7. The computer-readable medium of claim 6 , wherein the at least one neural network comprises a generator neural network and a discriminator neural network, and wherein the at least one neural network is configured to generate artificial images of the given class of object from the plurality of classes of objects based on the generator neural network and the discriminator neural network having been trained such that an error function was minimized.

8. The computer-readable medium of claim 6 , wherein the receptacle comprises a baggage item, and wherein the image of the receptacle comprises a real-world image of the baggage item captured by a real-world scanning device.

9. The computer-readable medium of claim 6 , wherein the program instructions that are executable by the at least one processor such that the computing system is configured to select the at least one neural network from the plurality of neural networks comprise instructions that are executable by the at least one processor such that the computing system is configured to: based on the image of the target object, make a determination that the target object belongs to the given class of object from the plurality of classes of objects; and based on the determination, select from the plurality of neural networks the at least one neural network that is configured to generate artificial images of the given class of object.

10. The computer-readable medium of claim 6 , wherein the instructions that are executable by the at least one processor further comprise instructions that, when executed by the at least one processor overlay the artificial image of the target object within the image of the receptacle with at least a portion of a background object from the image of the receptacle.

11. A computer-implemented method comprising:

receiving data associated with an input image comprising a target object of a given class from a plurality of classes of objects;

selecting at least one neural network from a plurality of neural networks, based on said data, to use to generate the artificial image of the target object, wherein the at least one neural network is selected based on whether it is most likely to generate an accurate simulated image of the target object relative to other ones of the plurality of the neural networks;

based on an image of a target object of a given class of object and the selected at least one neural network configured to generate artificial images of the given class of object, generating an artificial image of the target object that is substantially similar to real-world images of objects of the given class of objects captured by real-world scanning devices;

based on an image of a receptacle, selecting an insertion location within the receptacle in the image of the receptacle to insert the artificial image of the target object;

generating a combined image of the receptacle and the target object, wherein generating the combined image comprises inserting the artificial image of the target object into the image of the receptacle at the insertion location;

applying a variation to the combined image, wherein the variation comprises a change in at least one of an intensity, an obscuration, a noise, a magnification, a rotation, or a Z-effective encoding of the combined image; and

training one or more object detection algorithms with the combined image of the receptacle and the target object.

12. The computer-implemented method of claim 11 , wherein selecting the at least one neural network from the plurality of neural networks to use to generate the artificial image of the target object comprises: based on the image of the target object, making a determination that the target object belongs to the given class of object from the plurality of classes of objects; and based on the determination, selecting from the plurality of neural networks the at least one neural network that is configured to generate artificial images of the given class of object from the plurality of classes of objects.

13. The computer-implemented method of claim 11 , further comprising overlaying the artificial image of the target object within the image of the receptacle with at least a portion of a background object from the image of the receptacle.

14. The computing system of claim 1 wherein the instructions that are executable by the at least one processor further comprise instructions that, when executed, establish boundaries for the variation and randomly sample values within the boundaries of the variation.

15. The computer-readable medium of claim 6 , wherein the instructions that are executable by the at least one processor further comprise instructions that, when executed, establish boundaries for the variation and randomly sample values within the boundaries of the variation.

16. The computer-implemented method of claim 11 , further comprising establishing boundaries for the variation and randomly sampling values within the boundaries of the variation.

17. The computing system of claim 1 , wherein the input image is a projection image and the data is metadata.

18. The computer-implemented method of claim 6 , wherein the input image is a projection image and the data is metadata.

19. The computer-implemented method of claim 11 , wherein the input image is a projection image and the data is metadata.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 6, 2020
From: SYNAPSE TECHNOLOGY CORPORATION
To: RAPISCAN LABORATORIES, INC.
Reel/Frame 052322/0078 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 20, 2019
From: CINNAMON, IAN; FAVIERO, BRUNO BRASIL FERRARI; GAUTAM, SIMANTA
To: SYNAPSE TECHNOLOGY CORPORATION
Reel/Frame 051064/0039 →
Continuity (4)
Continuation 15799274 · Oct 31, 2017
Continuation 15727108 · Oct 6, 2017
Provisional Application 62547626 · Aug 18, 2017
Related Publication 20200051291A1 · Feb 13, 2020
Cited By (1)
US 12,385,854