Traffic signal understandings and representation for prediction, planning, and control

1. A method for vehicle prediction, planning, and control, comprising:

recording, using a monocular camera of an ego vehicle, visual traffic signal state information at an intersection, visual turn light and/or brake light state information of surrounding autonomous dynamic objects (ADOs), and traffic state information of the ego vehicle;

separately encoding the visual traffic signal state information, the visual turn light and/or brake light state information, and the traffic state information into corresponding traffic state latent spaces;

aggregating the corresponding traffic state latent spaces to form a generalized traffic geometry latent space;

interpreting the generalized traffic geometry latent space to form a traffic flow map including current and future vehicle trajectories;

decoding the generalized traffic geometry latent space to predict a vehicle behavior according to the traffic flow map based on the current and future vehicle trajectories; and

controlling the ego vehicle to follow a planned trajectory at the intersection in response to the predicted vehicle behavior and the current and future vehicle trajectories based on the traffic flow map without relying on map data or light detection and ranging (LIDAR) or radio detection and ranging (RADAR) sensors.

2. The method of claim 1 , in which separately encoding comprises:

encoding visual traffic signal state information at the intersection into a traffic signal state latent space of the corresponding traffic state latent spaces;

encoding turn light and/or brake light state information of surrounding autonomous dynamic objects (ADOs) into a turn/brake light signal state latent space of the corresponding traffic state latent spaces; and

encoding traffic state information of an ego vehicle into an ego state latent space of the corresponding traffic state latent spaces.

3. The method of claim 1 , in which aggregating the corresponding traffic state latent spaces comprises:

predicting a future traffic signal state according to a traffic light latent space of the corresponding traffic state latent spaces; and

predicting a vehicle state according to the future traffic signal state and a turn/brake light state latent space of the corresponding traffic state latent spaces.

4. The method of claim 1 , further comprising training a neural network to predict the vehicle behavior of a vehicle at the intersection using a traffic signal prediction as an input.

5. The method of claim 1 , in which generating the traffic flow map comprises:

aggregating vision inputs of traffic light signals at the intersection during operation of an ego vehicle;

predicting a future state of the traffic light signals at the intersection during operation of the ego vehicle;

processing turn/brake light signals of surrounding autonomous dynamic objects (ADOs) into a turn/brake light state latent space of the corresponding latent spaces;

predicting ADO vehicle behavior at the intersection according to the future state of the traffic light signals and the turn/brake light state latent space; and

generating the traffic flow map according to an observed ADO vehicle behavior and the predicted ADO vehicle behavior at the intersection.

6. The method of claim 1 , further comprising:

decoding the generalized traffic geometry latent space; and

predicting an intention of a vehicle at the intersection.

7. The method of claim 1 , further comprising:

decoding the generalized traffic geometry latent space; and

forecasting a trajectory of a vehicle at the intersection.

8. The method of claim 1 , further comprising planning the planned trajectory of the ego vehicle approaching the intersection according to the vehicle behavior and the traffic flow map based on the current and future vehicle trajectories.

9. A non-transitory computer-readable medium having program code recorded thereon for vehicle prediction, planning, and control, the program code being executed by a processor and comprising:

program code to record, using a monocular camera of an ego vehicle, visual traffic signal state information at an intersection, visual turn light and/or brake light state information of surrounding autonomous dynamic objects (ADOs), and traffic state information of an ego vehicle;

program code to separately encode the visual traffic signal state information, the visual turn light and/or brake light state information, and the traffic state information into corresponding traffic state latent spaces;

program code to aggregate the corresponding traffic state latent spaces to form a generalized traffic geometry latent space;

program code to interpret the generalized traffic geometry latent space to form a traffic flow map including current and future vehicle trajectories;

program code to decode the generalized traffic geometry latent space to predict a vehicle behavior according to the traffic flow map based on the current and future vehicle trajectories; and

program code to control the ego vehicle to follow a planned trajectory at the intersection in response to the predicted vehicle behavior and the current and future vehicle trajectories based on the traffic flow map without relying on map data or light detection and ranging (LIDAR) or radio detection and ranging (RADAR) sensors.

10. The non-transitory computer-readable medium of claim 9 , in which the program code to separately encode comprises:

program code to encode visual traffic signal state information at the intersection into a traffic signal state latent space of the corresponding traffic state latent spaces;

program code to encode turn light and/or brake light state information of surrounding autonomous dynamic objects (ADOs) into a turn/brake light signal state latent space of the corresponding traffic state latent spaces; and

program code to encode traffic state information of the ego vehicle into an ego state latent space of the corresponding traffic state latent spaces.

11. The non-transitory computer-readable medium of claim 9 , in which the program code to aggregate the corresponding traffic state latent spaces comprises:

program code to predict a future traffic signal state according to a traffic light latent space of the corresponding traffic state latent spaces; and

program code to predict a vehicle state according to the future traffic signal state and a turn/brake light state latent space of the corresponding traffic state latent spaces.

12. The non-transitory computer-readable medium of claim 9 , further comprising program code to train a neural network to predict the vehicle behavior of a vehicle at the intersection using a traffic signal prediction as an input.

13. The non-transitory computer-readable medium of claim 9 , in which generating the traffic flow map comprises:

program code to aggregate vision inputs of traffic light signals at the intersection during operation of the ego vehicle;

program code to predict a future state of the traffic light signals at the intersection during operation of the ego vehicle;

program code to process turn/brake light signals of surrounding autonomous dynamic objects (ADOs) into a turn/brake light state latent space of the corresponding latent spaces;

program code to predict ADO vehicle behavior at the intersection according to the future state of the traffic light signals and the turn/brake light state latent space; and

program code to generate the traffic flow map according to an observed ADO vehicle behavior and the predicted ADO vehicle behavior at the intersection.

14. The non-transitory computer-readable medium of claim 9 , further comprising:

program code to decode the generalized traffic geometry latent space; and

program code to predict an intention of a vehicle at the intersection.

15. The non-transitory computer-readable medium of claim 9 , further comprising:

program code to decode the generalized traffic geometry latent space; and

program code to forecast a trajectory of a vehicle at the intersection.

16. The non-transitory computer-readable medium of claim 9 , further comprising program code to plan the planned trajectory of the ego vehicle approaching the intersection according to the vehicle behavior and the traffic flow map based on the current and future vehicle trajectories.

17. A system for vehicle prediction, planning, and control, the system comprising:

a traffic state latent space model to record, using a monocular camera of an ego vehicle, visual traffic signal state information at an intersection, visual turn light and/or brake light state information of surrounding autonomous dynamic objects (ADOs), and traffic state information of the ego vehicle, and to separately encode the visual traffic signal state information, the visual turn light and/or brake light state information, and the traffic state information into corresponding traffic state latent spaces;

a traffic geometry latent space model to aggregate the corresponding traffic state latent spaces to form a generalized traffic geometry latent space;

a semantic traffic flow generation model to interpret the generalized traffic geometry latent space to form a traffic flow map including current and future vehicle trajectories;

a vehicle behavior prediction model to decode the generalized traffic geometry latent space to predict a vehicle behavior according to the traffic flow map based on the current and future vehicle trajectories; and

a controller module to control the ego vehicle to follow a planned trajectory at the intersection in response to the predicted vehicle behavior and the current and future vehicle trajectories based on the traffic flow map without relying on map data or light detection and ranging (LIDAR) or radio detection and ranging (RADAR) sensors.

18. The system of claim 17 , in which the vehicle behavior prediction model is further to decode the generalized traffic geometry latent space, and program code to predict an intention of a vehicle at the intersection.

19. The system of claim 17 , in which the vehicle behavior prediction model is further to decode the generalized traffic geometry latent space, and to forecast a trajectory of a vehicle at the intersection.

20. The system of claim 17 , further comprising a planner module to plan the planned trajectory of the ego vehicle approaching the intersection according to the vehicle behavior and the traffic flow map based on the current and future vehicle trajectories.