All weather traffic monitoring with roadside infrastructure
The disclosed system and methodology enable coexistence of networking and sensing on next-generation millimeter-wave (mmWave) picocells for traffic monitoring and pedestrian safety at intersections in all weather conditions. Existing wireless signal-based object detection systems suffer from limited resolution, and their outputs may not provide sufficient discriminatory information in complex scenes, such as traffic intersections. The disclosed system uses 5G picocells, which operate at mmWave frequency bands and provide higher data rates and higher sensing resolution than traditional wireless technology. It is difficult to run sensing applications and data transfer simultaneously on mmWave devices due to potential interference. MmWave devices are vulnerable to weak reflectivity and specularity challenges which may result in loss of information about objects and pedestrians. To address such challenges, the disclosed design uses customized deep learning models that not only can recover missing information about the target scene but also enable coexistence of networking and sensing.
1 . A Method for vehicle and pedestrian monitoring for safety at traffic intersections in all weather conditions, comprising:
providing a plurality of 5g picocells, operating at mmWave frequency bands, and associated with at least one target traffic intersection scene to be monitored;
obtaining reflected mmWave data from the plurality of picocells associated with the at least one target scene;
and inputting the obtained reflected mmWave data into a deep learning model trained to recover missing data about the target scene from the obtained reflected mmWave data while enabling coexistence of networking and sensing by the picocells
wherein the trained deep learning model uses context-aware object detection with a conditional Generative Adversarial Network (cGAN) to recover sensing samples otherwise intermittently unavailable;
wherein the coexistence of the networking and sensing is enabled by using “slot sequence” (S timestamps ) from the 5G network protocol.
2 . The method according to claim 1 , further comprising conducting collision prevention through at least one of warning pedestrian traffic and vehicle control, based on model output.
3 . The method according to claim 1 , wherein the obtained reflected mmWave data relates to at least one of real-time pedestrian and vehicular traffic at intersections, their speeds, vehicle proximity to intersection stop bars, occupied lanes, and vehicle types, to enable traffic monitoring and management.
4 . The method according to claim 1 , further comprising fully illuminating the at least one target scene by transmitting wireless signals and receiving them from reflecting off of different plural objects within the at least one target scene, to map the entire environment including static and dynamic objects within the at least one target scene, regardless of visibility or weather conditions.
5 . The method according to claim 1 , wherein the trained deep learning model uses the conditional Generative Adversarial Network (cGAN), which has multiple 2D convolution layers, 2D deconvolution layers, and skip connections on its Generator network to produce 2D bounding box depth images from mmWave heatmaps.
6 . The method according to claim 5 , further including conducting object detection with the trained deep learning model by using all data samples to first conduct dynamic heatmap recovery from missing data samples, and then conduct final object detection including use of recovered frames.
7 . The method according to claim 1 , wherein the deep learning model comprises a deep learning network with an encoder and decoder that converts the mmWave heatmaps into 2D bounding boxes with depth values.
8 . The method according to claim 1 , wherein the trained deep learning model uses the conditional Generative Adversarial Network (cGAN), which has a base model comprised of Generator (G) and Discriminator (D) function.
9 . The method according to claim 8 , further comprising providing two respective base models with same network architecture for pedestrians and vehicles, respectively.
10 . The method according to claim 9 , further including training the two respective base models including updating network parameters of each model with mmWave heatmaps and corresponding ground-truth depth images of pedestrians and vehicles, respectively.
11 . The method according to claim 8 , wherein the Discriminator (D) has two encoders used during training of the model for extracting features from mmWave heatmaps and ground-truth depth images, respectively.
12 . The method according to claim 11 , wherein the Discriminator (D) operates during training of the model to guide Generator (G) to learn to use mmWave heatmaps to generate output close to ground-truth depth images.
13 . The method according to claim 12 , wherein Discriminator (D) guides Generator (G) during training by generating an output probability value between 0 and 1 with a sigmoid activation function on its output layer, to indicate the closeness of an input depth image to a corresponding ground truth depth image.
14 . The method according to claim 13 , wherein training is completed whenever Discriminator (D) continuously outputs a probability value close to 0.5 for all the samples, reflecting that Discriminator (D) can no longer distinguish between ground truth and generated samples.
15 . The method according to claim 8 , wherein during training, mmWave sensing samples are dropped to emulate networking slots, and the majority of the time slots are assigned to network communication since the picocell's primary function is to support required data throughput.
16 . The method according to claim 15 , wherein during static and dynamic heatmaps are removed and recovered based on different slot sequence configurations.
17 . The method according to claim 1 , wherein the deep learning architecture predicts dynamic heatmap data, using previous dynamic heatmap data and the dominant reflecting points along with their corresponding velocities to forecast the movements of objects in future dynamic heatmaps.
18 . The method according to claim 1 , wherein the cGAN-based model comprises a learning framework having a network loss function which is a combination of Mean Squared Error (MSE) used to enforce pixel-to-pixel mapping in reconstructed depth images and Binary Cross Entropy (BCE) to measure the entropy loss of the Discriminator's output, focus on learning to produce accurate bounding boxes and correct depth values of objects rather than focusing on quality of generated image.
19 . A system for vehicle and pedestrian monitoring for safety at traffic intersections in all weather conditions, comprising:
a deep-learning model trained with reflected mmWave data from a plurality of picocells associated with at least one target intersection, to recover missing data about the target scene from the obtained reflected mmWave data;
one or more processors; and
one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
operating at mmWave frequency bands a plurality of 5g picocells, associated with the at least one target traffic intersection to be monitored;
and conducting collision prevention through at least one of warning pedestrian traffic and vehicle control, based on model output;
wherein the operations further comprise using context-aware object detection with a conditional Generative Adversarial Network (cGAN) to recover sensing samples otherwise intermittently unavailable.
20 . The system according to claim 19 , wherein the operations further comprise conducting object detection with the trained deep learning model by using all data samples to first conduct dynamic heatmap recovery from missing data samples, and then conduct final object detection including use of recovered frames.
21 . The system according to claim 20 , wherein the operations further comprise: establishing a base model comprised of Generator (G) and Discriminator (D) functions; and providing two respective base models with same network architecture for pedestrians and vehicles, respectively.
22 . The system according to claim 21 , wherein the operations further comprise: training the two respective base models including updating network parameters of each model with mmWave heatmaps and corresponding ground-truth depth images of pedestrians and vehicles, respectively; and providing each respective base model with an encoder and decoder that converts the mmWave heatmaps into 2D bounding boxes with depth values.
23 . The system according to claim 22 , wherein the operations further comprise operating the Discriminator (D) for training of the model by guiding Generator (G) to learn to use mmWave heatmaps to generate output close to ground-truth depth images.