Multi-task perception network with applications to scene understanding and advanced driver-assistance system
A method is provided in an Advanced Driver-Assistance System (ADAS). The method extracts, from an input video stream including a plurality of images using a multi-task Convolutional Neural Network (CNN), shared features across different perception tasks. The perception tasks include object detection and other perception tasks. The method concurrently solves, using the multi-task CNN, the different perception tasks in a single pass by concurrently processing corresponding ones of the shared features by respective different branches of the multi-task CNN to provide a plurality of different perception task outputs. Each respective different branch corresponds to a respective one of the different perception tasks. The method forms a parametric representation of a driving scene as at least one top-view map responsive to the plurality of different perception task outputs. The method controls an operation of the vehicle for collision avoidance responsive to the at least one top-view map indicating an impending collision.
1. A computer-implemented method in an Advanced Driver-Assistance System (ADAS), comprising: extracting, by a hardware processor from an input video stream comprising a plurality of images using a multi-task Convolutional Neural Network (CNN), shared features across different perception tasks, the different perception tasks comprising object detection and other perception tasks; concurrently solving, by the hardware processor using the multi-task CNN, the different perception tasks in a single pass by concurrently processing corresponding ones of the shared features by respective different branches of the multi-task CNN to provide a plurality of different perception task outputs, each of the respective different branches corresponding to a respective one of the different perception tasks; forming a parametric representation of a driving scene as at least one top-view map responsive to the plurality of different perception task outputs; and controlling an operation of a vehicle for collision avoidance responsive to the at least one top-view map indicating an impending collision.
2. The computer-implemented method of claim 1 , wherein the other perception tasks comprise semantic segmentation, depth estimation, and occlusion reasoning.
3. The computer-implemented method of claim 1 , wherein the hardware processor consists of a single GPU.
4. The computer-implemented method of claim 1 , further comprising:
associating bounding boxes across different pictures to obtain object tracks;
processing the bounding boxes to determine camera pose; and
localizing objects encapsulated by the bounding boxes responsive to the object tracks and the camera pose to provide refined object tracks for forming the at least one top-view map.
5. The computer-implemented method of claim 4 , wherein the refined object tracks are provided to be consistent over a given time period.
6. The computer-implemented method of claim 4 , further comprising generating a confidence score for each of the bounding boxes, and wherein the confidence score is used to obtain the object tracks.
7. The computer-implemented method of claim 1 , wherein the multi-task CNN comprises a plurality of sub-CNNs, each of the plurality of sub-CNNs for processing a respective different one of the different perception tasks.
8. The computer-implemented method of claim 1 , further comprising training the multi-task CNN using training data comprising two-dimensional object boxes, three-dimensional object boxes, sparse three-dimensional points, and semantic pixels.
9. The computer-implemented method of claim 8 , wherein the training data is annotated for a respective of the different perception tasks.
10. The computer-implemented method of claim 1 , wherein each of the semantic pixels are assigned to one of a plurality of available semantic categories.
11. The computer-implemented method of claim 1 , further comprising forming the top-view map using occlusion reasoning, the occlusion reasoning estimating semantics and distances for any pixels in the input video stream occluded by foreground objects.
12. The computer-implemented method of claim 11 , wherein occluded areas of a scene in a frame of the input video stream are unannotated for the occlusion reasoning.
13. The computer-implemented method of claim 1 , wherein the collision avoidance comprises controlling a vehicle input selected from a group consisting of braking and steering.
14. The computer-implemented method of claim 1 , further comprising performing a scene understanding task responsive to the top-view map to aid in the collision avoidance.
15. A computer program product for advanced driver-assistance, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising: extracting, by a hardware processor of the computer from an input video stream comprising a plurality of images using a multi-task Convolutional Neural Network (CNN), shared features across different perception tasks, the different perception tasks comprising object detection and other perception tasks; concurrently solving, using the multi-task CNN, the different perception tasks in a single pass by concurrently processing corresponding ones of the shared features by respective different branches of the multi-task CNN to provide a plurality of different perception task outputs, each of the respective different branches corresponding to a respective one of the different perception tasks; forming a parametric representation of a driving scene as at least one top-view map responsive to the plurality of different perception task outputs; and controlling an operation of a vehicle for collision avoidance responsive to the at least one top-view map indicating an impending collision.
16. The computer program product of claim 15 , wherein the other perception tasks comprise semantic segmentation, depth estimation, and occlusion reasoning.
17. The computer program product of claim 15 , wherein the hardware processor consists of a single GPU.
18. The computer program product of claim 15 , further comprising:
associating bounding boxes across different pictures to obtain object tracks;
processing the bounding boxes to determine camera pose; and
localizing objects encapsulated by the bounding boxes responsive to the object tracks and the camera pose to provide refined object tracks for forming the at least one top-view map.
19. The computer program product of claim 15 , wherein the multi-task CNN comprises a plurality of sub-CNNs, each of the plurality of sub-CNNs for processing a respective different one of the different perception tasks.
20. A computer processing system for advanced driver-assistance, comprising: a memory device including program code stored thereon; a hardware processor, operatively coupled to the memory device, and configured to run the program code stored on the memory device to extract, from an input video stream comprising a plurality of images using a multi-task Convolutional Neural Network (CNN), shared features across different perception tasks, the different perception tasks comprising object detection and other perception tasks; concurrently solve, using the multi-task CNN, the different perception tasks in a single pass by concurrently processing corresponding ones of the shared features by respective different branches of the multi-task CNN to provide a plurality of different perception task outputs, each of the respective different branches corresponding to a respective one of the different perception tasks; form a parametric representation of a driving scene as at least one top-view map responsive to the plurality of different perception task outputs; and control an operation of a vehicle for collision avoidance responsive to the at least one top-view map indicating an impending collision.