Object detection in driver assistance system

1 . A computer-implemented method for detecting objects within an advanced driver assistance system (ADAS), the method comprising:

obtaining road scene datasets from a plurality of cameras, including at least road scene images and road scene data annotations, to be provided to an object detection neural network communicating with an open-vocabulary detector of an autonomous vehicle;

converting, by a text prompter, the road scene data annotations into natural text inputs;

converting, by a text embedder, the natural text inputs into embeddings;

minimizing objective functions during training to adjust parameters of the object detection neural network, including a weak alignment objective function that determines a soft assignment of similarity between object detections from the road scene images and tokens from road scene captions, and a global alignment objective function that maximizes a similarity between feature embeddings that summarize the road scene images and the road scene captions;

analyzing a text description input that describes an entity from an input road scene obtained by the plurality of cameras;

detecting, by the object detection neural network, objects within the road scene datasets to provide alerts or notifications to the autonomous vehicle pertaining to the detected objects including the entity; and

generating a trajectory that controls the autonomous vehicle to navigate towards the entity within the input road scene.

2 . The computer-implemented method of claim 1 , wherein a reasoning logic queries the open-vocabulary detector with free-form text descriptions of the objects to understand the road scene datasets.

3 . The computer-implemented method of claim 1 , wherein each object is described with a bounding box and some semantic description.

4 . The computer-implemented method of claim 1 , wherein localization is applied to minimize localization ability of the object detection neural network.

5 . The computer-implemented method of claim 1 , wherein a classification loss is applied when category annotations are available for each object.

6 . The computer-implemented method of claim 1 , wherein the weak alignment objective function is trained from image-caption pairs without localization annotation.

7 . A computer program product for detecting objects within an advanced driver assistance system (ADAS), the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

obtaining road scene datasets from a plurality of cameras, including at least road scene images and road scene data annotations, to be provided to an object detection neural network communicating with an open-vocabulary detector of an autonomous vehicle;

converting, by a text prompter, the road scene data annotations into natural text inputs;

converting, by a text embedder, the natural text inputs into embeddings;

minimizing objective functions during training to adjust parameters of the object detection neural network, including a weak alignment objective function that determines a soft assignment of similarity between object detections from the road scene images and tokens from road scene captions, and a global alignment objective function that maximizes a similarity between feature embeddings that summarize the road scene images and the road scene captions;

analyzing a text description input that describes an entity from an input road scene obtained by the plurality of cameras;

detecting, by the object detection neural network, objects within the road scene datasets to provide alerts or notifications to the autonomous vehicle pertaining to the detected objects including the entity; and

generating a trajectory that controls the autonomous vehicle to navigate towards the entity within the input road scene.

8 . The computer program product of claim 7 , wherein a reasoning logic queries the open-vocabulary detector with free-form text descriptions of the objects to understand the road scene datasets.

9 . The computer program product of claim 7 , wherein each object is described with a bounding box and some semantic description.

10 . The computer program product of claim 7 , wherein localization is applied to minimize localization ability of the object detection neural network.

11 . The computer program product of claim 7 , wherein a classification loss is applied when category annotations are available for each object.

12 . The computer program product of claim 7 , wherein the weak alignment objective function is trained from image-caption pairs without localization annotation.

13 . A computer processing system for detecting objects within an advanced driver assistance system (ADAS), comprising:

a memory device for storing program code; and

a processor device, operatively coupled to the memory device, for running the program code to:

obtain road scene datasets from a plurality of cameras, including at least road scene images and road scene data annotations, to be provided to an object detection neural network communicating with an open-vocabulary detector of an autonomous vehicle;

convert, by a text prompter, the road scene data annotations into natural text inputs;

convert, by a text embedder, the natural text inputs into embeddings;

minimize objective functions during training to adjust parameters of the object detection neural network, including a weak alignment objective function that determines a soft assignment of similarity between object detections from the road scene images and tokens from road scene captions, and a global alignment objective function that maximizes a similarity between feature embeddings that summarize the road scene images and the road scene captions;

analyze a text description input that describes an entity from an input road scene obtained by the plurality of cameras and

detect, by the object detection neural network, objects within the road scene datasets to provide alerts or notifications to the autonomous vehicle pertaining to the detected objects including the entity; and

generate a trajectory that controls the autonomous vehicle to navigate towards the entity within the input road scene.

14 . The computer processing system of claim 13 , wherein a reasoning logic queries the open-vocabulary detector with free-form text descriptions of the objects to understand the road scene datasets.

15 . The computer processing system of claim 13 , wherein each object is described with a bounding box and some semantic description.

16 . The computer processing system of claim 13 , wherein localization is applied to minimize localization ability of the object detection neural network.

17 . The computer processing system of claim 13 , wherein a classification loss is applied when category annotations are available for each object.