System for real-time object detection and recognition using both image and size features

1. An object recognition system, the system comprising:

one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when executed, the one or more processors perform operations of:

using an integral channel features (ICF) detector, extracting a candidate target region from an input image of a scene surrounding a platform, wherein the candidate target region has an associated original confidence score representing a candidate object, wherein the input image is divided into a plurality of horizontal bands and ground truth objects are put into a same number of groups based on whether a location of the ground truth objects in the input image is in a band, with the objects in each group being used to estimate mean and standard deviation of object height distribution in the input image;

generating a modified confidence score based on a location and height of detection of the candidate object;

classifying the candidate target regions based on the modified confidence score using a trained convolutional neural network (CNN) classifier, resulting in classified objects;

using a multi-target tracker, tracking the classified objects for final classification of each classified object as a target or non-target; and

if the classified object is a target, then alerting security personnel via a displayed visual alert.

2. The system as set forth in claim 1 , wherein the ICF detector computes channel feature vectors for image frames of a video, and wherein, for each image frame, an ICF classifier is applied at multiple image scales and across the entire image frame.

3. The system as set forth in claim 1 , wherein the CNN classifier is implemented as interacting software modules comprising a CNN interface and a CNN server, wherein the CNN interface displays results received from the CNN server.

4. The system as set forth in claim 1 , wherein the trained CNN is used for both electro-optical (EO) and infrared (IR) image classification.

5. An object recognition system, the system comprising:

one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when executed, the one or more processors perform operations of:

using an integral channel features (ICF) detector, extracting a candidate target region from an input image of a scene surrounding a platform, wherein the candidate target region has an associated original confidence score representing a candidate object;

generating a modified confidence score based on a location and height of detection of the candidate object;

classifying the candidate target regions based on the modified confidence score using a trained convolutional neural network (CNN) classifier, resulting in classified objects;

using a multi-target tracker, tracking the classified objects for final classification of each classified object as a target or non-target; and

if the classified object is a target, then alerting security personnel via a displayed visual alert; and

wherein generating the modified confidence score uses a weighted Gaussian according to the following equation:

wf

=

exp

(

-

(

h

-

m

)

2

2

⁢

N

⁢

⁢

σ

)

,

and

modified

⁢

⁢

confidence

⁢

⁢

score

=

original

⁢

⁢

confidence

⁢

⁢

score

*

wf

,

wherein h denotes a height of the candidate object in the input image, m and σ denote mean and standard deviation, respectively, of object height distribution in the input image and bin, exp(.) denotes an exponential function, N is a multiplier and * denotes multiplication.

6. An object recognition system, the system comprising:

one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when executed, the one or more processors perform operations of:

using an integral channel features (ICF) detector, extracting a candidate target region from an input image of a scene surrounding a platform, wherein the candidate target region has an associated original confidence score representing a candidate object;

generating a modified confidence score based on a location and height of detection of the candidate object;

classifying the candidate target regions based on the modified confidence score using a trained convolutional neural network (CNN) classifier, resulting in classified objects;

using a multi-target tracker, tracking the classified objects for final classification of each classified object as a target or non-target; and

if the classified object is a target, then alerting security personnel via a displayed visual alert; and

wherein generating the modified confidence score uses a weighted gate according to the following equation:

wf

=

{

1

,

if

⁢

⁢



h

-

m



<

N

⁢

⁢

σ

0

,

otherwise

,

and

⁢

⁢

modified

⁢

⁢

confidence

⁢

⁢

score

=

original

⁢

⁢

confidence

⁢

⁢

score

*

wf

,

wherein h denotes a height of the candidate object in the input image, m and σ denote mean and standard deviation, respectively, of object height distribution in the input image and bin, N is a multiplier and * denotes multiplication.

7. The system as set forth in claim 1 , further comprising operations of:

classifying the candidate target regions based on the modified confidence score using a modified convolution network (CNN-2) classifier, resulting in modified classified objects; and

fusing the modified classified objects with the classified objects from the trained CNN classifier for processing by the multi-target tracker.

8. A computer program product for object recognition the computer program product comprising:

a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions by one or more processors, the one or more processors perform operations of:

using an integral channel features (ICF) detector, extracting a candidate target region from an input image of a scene surrounding a platform, wherein the candidate target region has an associated original confidence score representing a candidate object, wherein the input image is divided into a plurality of horizontal bands and ground truth objects are put into a same number of groups based on whether a location of the ground truth objects in the input image is in a band, with the objects in each group being used to estimate mean and standard deviation of object height distribution in the input image;

generating a modified confidence score based on a location and height of detection of the candidate object;

classifying the candidate target regions based on the modified confidence score using a trained convolutional neural network (CNN) classifier, resulting in classified objects;

using a multi-target tracker, tracking the classified objects for final classification of each classified object as a target or non-target; and

if the classified object is a target, then controlling a device based on the target.

9. The computer program product as set forth in claim 8 , wherein the ICF detector computes channel feature vectors for image frames of a video, and wherein, for each image frame, an ICF classifier is applied at multiple image scales and across the entire image frame.

10. The computer program product as set forth in claim 8 , wherein the CNN classifier is implemented as interacting software modules comprising a CNN interface and a CNN server, wherein the CNN interface displays results received from the CNN server.

11. The computer program product as set forth in claim 8 , wherein the trained CNN is used for both electro-optical (EO) and infrared (IR) image classification.

12. The computer program product as set forth in claim 8 , wherein generating the modified confidence score uses a weighted Gaussian according to the following equation:

wf

=

exp

(

-

(

h

-

m

)

2

2

⁢

N

⁢

⁢

σ

)

,

and

modified

⁢

⁢

confidence

⁢

⁢

score

=

original

⁢

⁢

confidence

⁢

⁢

score

*

wf

,

wherein h denotes a height of the candidate object in the input image, m and σ denote mean and standard deviation, respectively, of object height distribution in the input image and bin, exp(.) denotes an exponential function, N is a multiplier and * denotes multiplication.

13. The computer program product as set forth in claim 8 , wherein generating the modified confidence score uses a weighted gate according to the following equation:

wf

=

{

1

,

if

⁢

⁢



h

-

m



<

N

⁢

⁢

σ

0

,

otherwise

,

and

⁢

⁢

modified

⁢

⁢

confidence

⁢

⁢

score

=

original

⁢

⁢

confidence

⁢

⁢

score

*

wf

,

wherein h denotes a height of the candidate object in the input image, m and σ denote mean and standard deviation, respectively, of object height distribution in the input image and bin, N is a multiplier and * denotes multiplication.

14. The computer program product as set forth in claim 8 , further comprising operations of:

classifying the candidate target regions based on the modified confidence score using a modified convolution network (CNN-2) classifier, resulting in modified classified objects; and

fusing the modified classified objects with the classified objects from the trained CNN classifier for processing by the multi-target tracker.

15. A computer implemented method for object recognition, the method comprising an act of:

using an integral channel features (ICF) detector, extracting a candidate target region from an input image of a scene surrounding a platform, wherein the candidate target region has an associated original confidence score representing a candidate object, wherein the input image is divided into a plurality of horizontal bands and ground truth objects are put into a same number of groups based on whether a location of the ground truth objects in the input image is in a band, with the objects in each group being used to estimate mean and standard deviation of object height distribution in the input image;

generating a modified confidence score based on a location and height of detection of the candidate object;

classifying the candidate target regions based on the modified confidence score using a trained convolutional neural network (CNN) classifier, resulting in classified objects;

using a multi-target tracker, tracking the classified objects for final classification of each classified object as a target or non-target; and

if the classified object is a target, then controlling a device based on the target.

16. The method as set forth in claim 15 , wherein the ICF detector computes channel feature vectors for image frames of a video, and wherein, for each image frame, an ICF classifier is applied at multiple image scales and across the entire image frame.

17. The method as set forth in claim 15 , wherein the CNN classifier is implemented as interacting software modules comprising a CNN interface and a CNN server, wherein the CNN interface displays results received from the CNN server.

18. The method as set forth in claim 15 , wherein the trained CNN is used for both electro-optical (EO) and infrared (IR) image classification.

19. The method as set forth in claim 15 , wherein generating the modified confidence score uses a weighted Gaussian according to the following equation:

wf

=

exp

(

-

(

h

-

m

)

2

2

⁢

N

⁢

⁢

σ

)

,

and

modified

⁢

⁢

confidence

⁢

⁢

score

=

original

⁢

⁢

confidence

⁢

⁢

score

*

wf

,

wherein h denotes a height of the candidate object in the input image, m and σ denote mean and standard deviation, respectively, of object height distribution in the input image and bin, exp(.) denotes an exponential function, N is a multiplier and * denotes multiplication.

20. The method as set forth in claim 15 , wherein generating the modified confidence score uses a weighted gate according to the following equation:

wf

=

{

1

,

if

⁢

⁢



h

-

m



<

N

⁢

⁢

σ

0

,

otherwise

,

and

⁢

⁢

modified

⁢

⁢

confidence

⁢

⁢

score

=

original

⁢

⁢

confidence

⁢

⁢

score

*

wf

,

wherein h denotes a height of the candidate object in the input image, m and σ denote mean and standard deviation, respectively, of object height distribution in the input image and bin, N is a multiplier and * denotes multiplication.

21. The method as set forth in claim 15 , further comprising acts of:

classifying the candidate target regions based on the modified confidence score using a modified convolution network (CNN-2) classifier, resulting in modified classified objects; and

fusing the modified classified objects with the classified objects from the trained CNN classifier for processing by the multi-target tracker.