System and method for aerial video traffic analysis
A system and method for aerial video traffic analysis are disclosed. A particular embodiment is configured to: receive a captured video image sequence from an unmanned aerial vehicle (UAV); clip the video image sequence by removing unnecessary images; stabilize the video image sequence by choosing a reference image and adjusting other images to the reference image; extract a background image of the video image sequence for vehicle segmentation; perform vehicle segmentation to identify vehicles in the video image sequence on a pixel by pixel basis; determine a centroid, heading, and rectangular shape of each identified vehicle; perform vehicle tracking to detect a same identified vehicle in multiple image frames of the video image sequence; and produce output and visualization of the video image sequence including a combination of the background image and the images of each identified vehicle.
1. A system comprising:
an unmanned aerial vehicle (UAV), equipped with a camera, deployed at an elevated position at a monitored location, the UAV configured to capture a video image sequence of the monitored location for a pre-determined period of time using the UAV camera;
a data processor; and
an image processing module, executable by the data processor, the image processing module being configured to:
receive the captured video image sequence from the UAV;
clip the video image sequence by removing unnecessary images from the video image sequence;
stabilize the video image sequence by choosing a reference image and adjusting other images to the reference image;
extract a background image of the video image sequence for vehicle segmentation;
perform vehicle segmentation to identify vehicles in the video image sequence on a pixel by pixel basis using a trained neural network to classify individual pixels and produce a collection of pixel classifications corresponding to the video image sequence;
generate a vehicle segmentation mask to determine a general shape of each vehicle identified in the video image sequence, the vehicle segmentation mask being generated using the collection of pixel classifications produced by the trained neural network;
determine a centroid, heading, and rectangular shape of each identified vehicle based on the vehicle segmentation mask and the general shape of each vehicle;
perform vehicle tracking to detect a same identified vehicle in multiple image frames of the video image sequence; and
produce output and visualization of the video image sequence including a combination of the background image and the images of each identified vehicle.
2. The system of claim 1 wherein the image processing module being configured to extract a background image of the video image sequence by inferring a dominant color value from a collection of frames sampled over time from the video image sequence.
3. The system of claim 1 wherein the image processing module being configured to perform vehicle segmentation by concatenating a video image frame with the corresponding background image.
4. The system of claim 1 wherein the image processing module includes machine learnable components.
5. The system of claim 1 wherein the image processing module being configured to generate a direction along which a variance of the shape of a vehicle as a distribution is maximized.
6. The system of claim 1 wherein the image processing module being configured to determine if a vehicle detection overlaps in two sequential image frames.
7. The system of claim 1 wherein the output and visualization includes visual bounding boxes and velocity vectors for each identified vehicle.
8. The system of claim 1 further including a human driver model configured to predict or simulate human driver behaviors.
9. A method comprising:
receiving a captured video image sequence from an unmanned aerial vehicle (UAV);
clipping the video image sequence by removing unnecessary images fromthe video image sequence;
stabilizing the video image sequence by choosing a reference image and adjusting other images to the reference image;
extracting a background image of the video image sequence for vehicle segmentation;
performing vehicle segmentation to identify vehicles in the video image sequence on a pixel by pixel basis using a trained neural network to classify individual pixels and produce a collection of pixel classifications corresponding to the video image sequence;
generating a vehicle segmentation mask to determine a general shape of each vehicle identified in the video image sequence, the vehicle segmentation mask being generated using the collection of pixel classifications produced by the trained neural network;
determining a centroid, heading, and rectangular shape of each identified vehicle based on the vehicle segmentation mask and the general shape of each vehicle;
performing vehicle tracking to detect a same identified vehicle in multiple image frames of the video image sequence; and
producing output and visualization of the video image sequence including a combination of the background image and the images of each identified vehicle.
10. The method of claim 9 including extracting the background image of the video image sequence by inferring a dominant color value from a collection of frames sampled over time from the video image sequence.
11. The method of claim 9 including performing vehicle segmentation by concatenating a video image frame with the corresponding background image.
12. The method of claim 9 including using machine learnable components.
13. The method of claim 9 including generating a direction along which a variance of the shape of a vehicle as a distribution is maximized.
14. The method of claim 9 including determining if a vehicle detection overlaps in two sequential image frames.
15. The method of claim 9 wherein the output and visualization includes visual bounding boxes and velocity vectors for each identified vehicle.
16. The method of claim 9 further including providing a human driver model configured to predict or simulate human driver behaviors.