Method and system for detecting multiple moving objects from real-time aerial images

1. A method for detecting multiple objects from real-time images, comprising:

performing, using a central processing unit (CPU) host, an image segmentation process to divide real-time input images into a plurality of image partitions;

allocating a multitude of image memory according to the plurality of image partitions for parallel computation in a plurality of graphics processing units (GPUs);

performing, by registration kernels of the plurality of GPUs, a fast block-wise registration process to register the plurality of image partitions;

performing, by mask setting kernels of the plurality of GPUs, a mark setting process for the registered image partitions to stitch the registered images into output images;

performing, by background generation kernels of the plurality of GPUs, a background generation process based on the output images to generate background images through median filter;

performing, by foreground generation kernels of the plurality of GPUs, a foreground generation process based on the output images to generate foreground images based on a Hyper-Q computation framework;

performing, by classification kernels of the plurality of GPUs, a support vector machine classification process to classify a plurality of objects of the real-time input images; and

generating, by the CPU host, visualization classification images based on the background images, foreground images and the plurality of objects.

2. The method of claim 1 , wherein:

the real-time input images are generated from a Wide Area Motion Imagery system on an aerial platform;

a frame rate of the real-time input images is at least two frames per second; and

a scale of each real-time input image has a magnitude of million orders of pixels in resolution.

3. The method of claim 1 , wherein:

the fast block-wise registration process is a Compute Unified Device Architecture (CUDA) based parallel computing infrastructure, and comprises:

performing a block-wise speeded up robust features extraction process for each image partition;

performing a point matching process for each image partition;

using a random sample consensus algorithm to remove outlier points from the plurality of image partitions; and

performing a transformation estimation process of the image partitions to generate block-wise homography matrices.

4. The method of claim 1 , wherein each registration kernel is configured to have one computation node integrated with eight groups of four image partitions at a time instant.

5. The method of claim 3 , wherein the mark setting process comprises:

stitching portions of the registered image partitions based on the block-wise homography matrices generated from the transformation estimation process;

wherein a number of threads per block is consistent with available shared memory of the plurality of GPUs.

6. The method of claim 3 , wherein the point matching process is based on Brute-force (BF) or Fast Library for Approximate Nearest Neighbors (FLANN).

7. The method of claim 1 , wherein the background generation process:

comprises a background setting step for setting each image partition to a mask with zero pixel values, an image averaging step, and a background extraction step; and

is a parallelized process implemented based on the plurality of GPUs using data CUDA structure dim3.

8. The method of claim 1 , wherein the foreground generation process:

comprises a pixel value comparison step, a value assigning step, and a foreground extraction step;

implements the Hyper-Q computation infrastructure to enable multiple CPU cores to launch work on a single GPU simultaneously for increasing GPUs utilization and slashing CPU idle time; and

introduces Grid Management Unit to create multiple hardware work queues to reduce a synchronization time.

9. The method of claim 1 , wherein the support vector machine classification process comprises:

implementing histogram of oriented gradients (HOG) to compute color gradients and obtain gradient magnitudes and orientations via convolution; and

calculating probabilities or confidence levels of the multiple objects based on the gradient magnitudes and orientations.

10. The method of claim 1 , further comprising:

asynchronously transmitting the classified plurality of objects from the multiple GPUs to the CPU host to reduce or eliminate a synchronization time.

11. The method of claim 1 , further comprising:

generating a graphical user interface through an open source computer vision library associated with the CPU host for monitoring the multiple objects on the visualization classification images in real-time;

wherein the multiple objects include at least vehicles and people.

12. A system for detecting multiple objects from real-time images, comprising:

a CPU host configured for:

performing an image segmentation process to divide real-time input images into a plurality of image partitions, and

allocating a multitude of image memory according to the plurality of image partitions for parallel computation; and

a plurality of graphics processing units (GPUs) comprising:

registration kernels for performing a fast block-wise registration process to register the plurality of image partitions,

mask setting kernels for performing a mark setting process for the registered image partitions to stitch the registered images into output images,

background generation kernels for performing a background generation process based on the output images to generate background images through median filter,

foreground generation kernels for performing a foreground generation process based on the output images to generate foreground images based on a Hyper-Q computation infrastructure, and

classification kernels for performing a support vector machine classification process to classify a plurality of objects of the real-time input images;

wherein the CPU host is further configured for generating visualization classification images based on the background images, foreground images and the plurality of objects.

13. The system of claim 12 , wherein:

the real-time input images are generated from a Wide Area Motion Imagery system on an aerial platform;

a frame rate of the real-time input images is at least two frames per second;

a scale of each real-time input image has a magnitude of million orders of pixels in resolution; and

the multiple objects include at least vehicles and people.

14. The system of claim 12 , wherein:

the registration kernels are configured for performing the fast block-wise registration process based a Compute Unified Device Architecture (CUDA) based parallel computing infrastructure, by:

performing a block-wise speeded up robust features extraction process for each image partition;

performing a point matching process for each image partition;

using a random sample consensus (RANSAC) algorithm to remove outlier points from the plurality of image partitions; and

performing a transformation estimation process of the image partitions to generate block-wise homography matrices;

wherein each registration kernel is configured to have one cluster integrated with eight groups of four image partitions at a time instant.

15. The system of claim 14 , wherein the mark setting kernels are configured for:

stitching portions of the registered image partitions based on the block-wise homography matrices generated from the transformation estimation process;

wherein a number of threads per block is consistent with available shared memory of the plurality of GPUs.

16. The system of claim 12 , wherein the background generation kernels are configured for:

performing a background setting step for setting each image partition to a mask with zero pixel values, an image averaging step, and a background extraction step; and

implementing a parallelized process based on the plurality of GPUs using CUDA data structure dim3.

17. The system of claim 12 , wherein the foreground generation kernels are configured for:

performing a pixel value comparison step, a value assigning step, and a foreground extraction step;

implementing the Hyper-Q computation framework to enable multiple CPU cores to launch work on a single GPU simultaneously for increasing GPUs utilization and slashing CPU idle time; and

introducing Grid Management Unit to create multiple hardware work queues to reduce a synchronization time.

18. The system of claim 12 , wherein the classification kernels are configured for:

implementing histogram of oriented gradients to compute color gradients and obtain gradient magnitudes and orientations via convolution; and

calculating probabilities or confidence levels of the multiple objects based on the gradient magnitudes and orientations.

19. The system of claim 12 , wherein:

the classified plurality of objects are asynchronously transmitted from the multiple GPUs to the CPU host to reduce or eliminate a synchronization time.

20. The system of claim 12 , wherein the CPU host is further configured for:

generating a graphical user interface through an open source computer vision library for monitoring the multiple objects on the visualization classification images in real-time.