IP Library Granted Patent US 11,526,710
Granted Patent B1
US 11,526,710 · App. 16/825,901 · Granted Dec 13, 2022

Systems and methods for promissory image classification

Inventor: Daniel Clark Salo (Durham, NC)
Assignee: Proofpoint, Inc.
G06K9/6284G06K9/627G06K9/6256G06V30/40
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,526,710
App. No.
16/825,901
Granted
Dec 13, 2022
Kind
B1
Abstract

Systems, methods and products for classifying images according to a visual concept where, in one embodiment, a system includes an object detector and a visual concept classifier, the object detector being configured to detect objects depicted in an image and generate a corresponding object data set identifying the objects and containing information associated with each of the objects, the visual concept classifier being configured to examine the object data set generated by the object detector, detect combinations of the information in the object data set that are high-precision indicators of the designated visual concept being contained in the image, generate a classification for the object data set with respect to the designated visual concept, and associate the classification with the image, wherein the classification identifies the image as either containing the designated visual concept or not containing the designated visual concept.

Claims (79)

1. A method comprising:

for each of one or more images, generating a visual concept classification of each image by:

training an object detector to detect the one or more objects, the object detector comprising a convolutional neural network, the training including

selecting a set of training images, wherein each training image contains one or more objects associated with the designated visual concept, wherein each training image in a first subset of the training images contains the designated visual concept, and wherein each training image in a second subset of the training images does not contain the designated visual concept, and

for each of the images in the set of training images,

generating an object data set identifying one or more objects contained in the image and one or more pieces of information associated with each of the one or more objects,

providing the image to the object detector as an input and providing the corresponding object data set to the object detector as an expected output, and

processing the image and the corresponding object data set by the object detector, thereby training the object detector to detect the corresponding object data set in the image;

examining the image and detecting a set of objects depicted therein;

generating a set of object data corresponding to the objects depicted in the image, wherein the set of object data includes one or more pieces of information associated with each object in the set of objects;

examining the set of object data and generating a classification for the set of object data with respect to a designated visual concept; and

associating the classification with the image, wherein the classification identifies the image as either containing the designated visual concept or not containing the designated visual concept.

2. The method of claim 1 , wherein the designated visual concept comprises a corresponding fiduciary promise.

3. The method of claim 2 , further comprising, for each of one or more images, when the image is classified as containing the corresponding fiduciary promise, flagging the image and providing the flagged image to a user interface that enables manual review of the image by a user.

4. The method of claim 1 , wherein the first set of object data includes, for each of the one or more detected objects: an object type; a location; a size, and a confidence level.

5. The method of claim 4 , wherein the first set of object data includes one or more metadata items that are associated with the image prior to detecting the one or more objects depicted in the image.

6. The method of claim 1 , further comprising training a visual concept classifier to classify the object data set, the visual concept classifier comprising a machine learning engine, the training comprising,

for each of the images in the set of training images,

generating a classification for the image with respect to the designated visual concept,

providing the object data set and the corresponding classification to the visual concept classifier, and

processing the object data set and the corresponding classifier by the visual concept classifier, thereby training the visual concept classifier to detect the corresponding classification from the object data set.

7. The method of claim 1 , further comprising:

receiving a first plurality of messages associated with a business enterprise;

identifying a second plurality of image-containing messages, wherein the second plurality of image-containing messages is a reduced subset of the first plurality of messages;

extracting from each message of the second plurality of messages at least one of the one or more images; and

designating a third plurality of messages containing the designated visual concept, wherein each message of the third plurality of messages contains one of the one or more images containing the designated visual concept, wherein the third plurality of messages is a reduced subset of the second plurality of messages.

8. A system for detecting a designated visual concept in an image,

the system comprising:

an object detector having a convolutional neural network which is trained to detect the one or more objects, the object detector configured to receive the image,

detect a set of objects depicted therein, and

generate an object data set corresponding to the objects depicted in the image, wherein the object data set includes one or more pieces of information associated with each object in the set of objects; and

a visual concept classifier configured to

receive the object data set from the object detector,

examine the object data set,

detect ones or combinations of the pieces of information in the object data set that are high-precision indicators of the designated visual concept being contained in the image,

generate a classification for the object data set with respect to the designated visual concept, and

associate the classification with the image, wherein the classification identifies the image as either containing the designated visual concept or not containing the designated visual concept,

wherein the object detector is trained by

selecting a set of training images, wherein each training image contains one or more objects associated with the designated visual concept, wherein each training image in a first subset of the training images contains the designated visual concept, and wherein each training image in a second subset of the training images does not contain the designated visual concept,

generating an object data set identifying one or more objects contained in the image and one or more pieces of information associated with each of the one or more objects,

providing the image to the object detector as an input and providing the corresponding object data set to the object detector as an expected output, and

processing the image and the corresponding object data set by the object detector, thereby training the object detector to detect the corresponding object data set in the image;

wherein the visual concept classifier comprises a machine learning engine which is trained to classify the object data set, the training comprising, for each of the images in the set of training images,

generating a classification for the image with respect to the designated visual concept,

providing the object data set and the corresponding classification to the visual concept classifier, and

processing the object data set and the corresponding classifier by the visual concept classifier, thereby training the visual concept classifier to detect the corresponding classification from the object data set.

9. The system of claim 8 , wherein the designated visual concept comprises a corresponding fiduciary promise, and wherein the visual concept classifier is further configured to, when the image is classified as containing the corresponding fiduciary promise, flag the image and provide the flagged image to a user interface that enables manual review of the image by a user.

10. The system of claim 8 , wherein the first set of object data includes, for each of the one or more detected objects: an object type; a location; a size, and a confidence level.

11. The system of claim 10 , wherein the first set of object data includes one or more metadata items that are associated with the image prior to detecting the one or more objects depicted in the image.

12. The system of claim 8 , wherein the system is configured to

receive a first plurality of messages associated with a business enterprise;

identify a second plurality of image-containing messages, wherein the second plurality of image-containing messages is a reduced subset of the first plurality of messages;

extract from each message of the second plurality of messages at least one of the one or more images; and

designate a third plurality of messages containing the designated visual concept, wherein each message of the third plurality of messages contains one of the one or more images containing the designated visual concept, wherein the third plurality of messages is a reduced subset of the second plurality of messages.

13. A computer program product comprising a non-transitory computer-readable medium storing instructions executable by one or more processors to perform:

training an object detector to detect the one or more objects, the object detector comprising a convolutional neural network, the training including

selecting a set of training images, wherein each training image contains one or more objects associated with the designated visual concept, wherein each training image in a first subset of the training images contains the designated visual concept, and wherein each training image in a second subset of the training images does not contain the designated visual concept, and

for each of the images in the set of training images,

generating an object data set identifying one or more objects contained in the image and one or more pieces of information associated with each of the one or more objects,

providing the image to the object detector as an input and providing the corresponding object data set to the object detector as an expected output, and

processing the image and the corresponding object data set by the object detector, thereby training the object detector to detect the corresponding object data set in the image; and

training a visual concept classifier to classify the object data set, the visual concept classifier comprising a machine learning engine, the training comprising,

for each of the images in the set of training images,

generating a classification for the image with respect to the designated visual concept,

providing the object data set and the corresponding classification to the visual concept classifier, and

processing the object data set and the corresponding classifier by the visual concept classifier, thereby training the visual concept classifier to detect the corresponding classification from the object data set;

for each of one or more images, generating a visual concept classification of each image by:

examining the image and detecting a set of objects depicted therein;

generating a set of object data corresponding to the objects depicted in the image, wherein the set of object data includes one or more pieces of information associated with each object in the set of objects;

examining the set of object data and generating a classification for the set of object data with respect to a designated visual concept; and

associating the classification with the image, wherein the classification identifies the image as either containing the designated visual concept or not containing the designated visual concept.

14. The computer program product of claim 13 , wherein the designated visual concept comprises a corresponding fiduciary promise; wherein the instructions are further executable by the one or more processors to perform, for each of one or more images, when the image is classified as containing the corresponding fiduciary promise, flagging the image and providing the flagged image to a user interface that enables manual review of the image by a user.

15. The computer program product of claim 13 , wherein the first set of object data includes, for each of the one or more detected objects: an object type; a location; a size, and a confidence level.

16. The computer program product of claim 15 , wherein the first set of object data includes one or more metadata items that are associated with the image prior to detecting the one or more objects depicted in the image.

17. The computer program product of claim 13 , wherein the instructions are further executable by the one or more processors to perform:

receiving a first plurality of messages associated with a business enterprise;

identifying a second plurality of image-containing messages, wherein the second plurality of image-containing messages is a reduced subset of the first plurality of messages;

extracting from each message of the second plurality of messages at least one of the one or more images; and

designating a third plurality of messages containing the designated visual concept, wherein each message of the third plurality of messages contains one of the one or more images containing the designated visual concept, wherein the third plurality of messages is a reduced subset of the second plurality of messages.

Assignments (5)
SECOND LIEN INTELLECTUAL PROPERTY SECURITY AGREEMENT Recorded Dec 8, 2025
From: PROOFPOINT, INC.
To: U.S. BANK TRUST COMPANY, NATIONAL ASSOCIATION, AS COLLATERAL AGENT
Reel/Frame 073889/0677 →
RELEASE OF SECOND LIEN SECURITY INTEREST IN INTELLECTUAL PROPERTY Recorded Mar 21, 2024
From: GOLDMAN SACHS BANK USA, AS AGENT
To: PROOFPOINT, INC.
Reel/Frame 066865/0648 →
FIRST LIEN INTELLECTUAL PROPERTY SECURITY AGREEMENT Recorded Aug 31, 2021
From: PROOFPOINT, INC.
To: GOLDMAN SACHS BANK USA, AS COLLATERAL AGENT
Reel/Frame 057389/0615 →
SECOND LIEN INTELLECTUAL PROPERTY SECURITY AGREEMENT Recorded Aug 31, 2021
From: PROOFPOINT, INC.
To: GOLDMAN SACHS BANK USA, AS COLLATERAL AGENT
Reel/Frame 057389/0642 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 12, 2020
From: SALO, DANIEL CLARK
To: PROOFPOINT, INC.
Reel/Frame 053471/0891 →