Intervention behavior prediction with continuous confounders

1 . A method comprising:

receiving scene data characterizing a scene that includes a first agent and an autonomous vehicle in an environment;

receiving intervention data specifying a planned intervention to be performed by the autonomous vehicle;

generating, using a conditional behavior prediction model, a conditional behavior prediction output that assigns, to each of a plurality of possible future behaviors, (i) a respective conditional likelihood that the first agent performs the possible future behavior given that the autonomous vehicle performs the planned intervention and (ii) a predicted value of a confounder variable for the possible future behavior;

generating an intervention behavior prediction for the first agent by, for each possible future behavior, generating a corrected likelihood for the possible future behavior based on the respective conditional likelihood for the possible future behavior and the predicted value of the confounder variable for the possible future behavior;

selecting a planned trajectory for the autonomous vehicle based on the intervention behavior prediction; and

causing the autonomous vehicle to operate based on the planned trajectory.

2 . The method of claim 1 , further comprising:

determining a predicted conditional probability that is conditioned on the planned intervention for each of the predicted values of the confounder variable; and

wherein generating the corrected likelihood for the possible future behavior comprises generating the corrected likelihood for the possible future behavior based on the respective conditional likelihood for the possible future behavior, the predicted value of the confounder variable for the possible future behavior, and the predicted conditional probability for each of the predicted values for the confounder variable.

3 . The method of claim 2 , wherein determining a predicted probability for each of the predicted values of the confounder variable comprises:

processing a confounder prediction input generated from the scene data and the intervention data using a confounder prediction model, wherein the confounder prediction model is configured to receive the confounder prediction input and to process the confounder prediction input to generate a confounder distribution over a plurality of possible values for the confounder variable, wherein the confounder distribution comprises a predicted probability value for each of the possible values of the confounder variable.

4 . The method of claim 3 , further comprising:

processing the scene data using an encoder neural network to generate an encoded representation of the scene data, wherein an input to the conditional behavior prediction model and the confounder prediction input are a same input that comprises the encoded representation and the intervention data.

5 . The method of claim 2 , further comprising:

determining a predicted marginal probability for each of the predicted values of the confounder variable; and

wherein generating the corrected likelihood for the possible future behavior comprises generating the corrected likelihood for the possible future behavior based on the respective conditional likelihood for the possible future behavior, the predicted value of the confounder variable for the possible future behavior, the predicted conditional probability for the predicted value for the confounder variable, and the predicted marginal probability for the predicted value for the confounder variable.

6 . The method of claim 1 , wherein the confounder variable comprises a reactivity of the first agent to the planned intervention.

7 . The method of claim 1 , wherein the first agent is a road-user that is in a same environment as the autonomous vehicle.

8 . The method of claim 1 , further comprising:

obtaining a plurality of additional planned interventions by the autonomous vehicle;

for each additional planned intervention of the plurality of planned interventions, computing a respective additional intervention behavior prediction for the first agent in reaction to the autonomous vehicle performing the planned intervention; and

wherein selecting a planned trajectory for the autonomous vehicle based on the intervention behavior prediction comprises selecting the planned trajectory for the autonomous vehicle based on the intervention behavior prediction and the additional intervention behavior predictions corresponding to the plurality of planned interventions.

9 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or computers to perform operations comprising:

receiving scene data characterizing a scene that includes a first agent and an autonomous vehicle in an environment;

receiving intervention data specifying a planned intervention to be performed by the autonomous vehicle;

generating, using a conditional behavior prediction model, a conditional behavior prediction output that assigns, to each of a plurality of possible future behaviors, (I) a respective conditional likelihood that the first agent performs the possible future behavior given that the autonomous vehicle performs the planned intervention and (ii) a predicted value of a confounder variable for the possible future behavior;

generating an intervention behavior prediction for the first agent by, for each possible future behavior, generating a corrected likelihood for the possible future behavior based on the respective conditional likelihood for the possible future behavior and the predicted value of the confounder variable for the possible future behavior;

selecting a planned trajectory for the autonomous vehicle based on the intervention behavior prediction; and

causing the autonomous vehicle to operate based on the planned trajectory.

10 . The system of claim 9 , wherein the operations further comprise:

determining a predicted conditional probability that is conditioned on the planned intervention for each of the predicted values of the confounder variable; and

wherein generating the corrected likelihood for the possible future behavior comprises generating the corrected likelihood for the possible future behavior based on the respective conditional likelihood for the possible future behavior, the predicted value of the confounder variable for the possible future behavior, and the predicted conditional probability for each of the predicted values for the confounder variable.

11 . The system of claim 10 , wherein determining a predicted probability for each of the predicted values of the confounder variable comprises:

processing a confounder prediction input generated from the scene data and the intervention data using a confounder prediction model, wherein the confounder prediction model is configured to receive the confounder prediction input and to process the confounder prediction input to generate a confounder distribution over a plurality of possible values for the confounder variable, wherein the confounder distribution comprises a predicted probability value for each of the possible values of the confounder variable.

12 . The system of claim 11 , wherein the operations further comprise:

processing the scene data using an encoder neural network to generate an encoded representation of the scene data, wherein an input to the conditional behavior prediction model and the confounder prediction input are a same input that comprises the encoded representation and the intervention data.

13 . The system of claim 10 , wherein the operations further comprise:

determining a predicted marginal probability for each of the predicted values of the confounder variable; and

wherein generating the corrected likelihood for the possible future behavior comprises generating the corrected likelihood for the possible future behavior based on the respective conditional likelihood for the possible future behavior, the predicted value of the confounder variable for the possible future behavior, the predicted conditional probability for the predicted value for the confounder variable, and the predicted marginal probability for the predicted value for the confounder variable.

14 . The system of claim 9 , wherein the confounder variable comprises a reactivity of the first agent to the planned intervention.

15 . The system of claim 9 , wherein the first agent is a road-user that is in a same environment as the autonomous vehicle.

16 . The system of claim 9 , wherein the operations further comprise:

obtaining a plurality of additional planned interventions by the autonomous vehicle;

for each additional planned intervention of the plurality of planned interventions, computing a respective additional intervention behavior prediction for the first agent in reaction to the autonomous vehicle performing the planned intervention; and

wherein selecting a planned trajectory for the autonomous vehicle based on the intervention behavior prediction comprises selecting the planned trajectory for the autonomous vehicle based on the intervention behavior prediction and the additional intervention behavior predictions corresponding to the plurality of planned interventions.

17 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving scene data characterizing a scene that includes a first agent and an autonomous vehicle in an environment;

receiving intervention data specifying a planned intervention to be performed by the autonomous vehicle;

generating, using a conditional behavior prediction model, a conditional behavior prediction output that assigns, to each of a plurality of possible future behaviors, (i) a respective conditional likelihood that the first agent performs the possible future behavior given that the autonomous vehicle performs the planned intervention and (ii) a predicted value of a confounder variable for the possible future behavior;

generating an intervention behavior prediction for the first agent by, for each possible future behavior, generating a corrected likelihood for the possible future behavior based on the respective conditional likelihood for the possible future behavior and the predicted value of the confounder variable for the possible future behavior;

selecting a planned trajectory for the autonomous vehicle based on the intervention behavior prediction; and

causing the autonomous vehicle to operate based on the planned trajectory.

18 . The non-transitory computer storage media of claim 17 , wherein the operations further comprise:

determining a predicted conditional probability that is conditioned on the planned intervention for each of the predicted values of the confounder variable; and

wherein generating the corrected likelihood for the possible future behavior comprises generating the corrected likelihood for the possible future behavior based on the respective conditional likelihood for the possible future behavior, the predicted value of the confounder variable for the possible future behavior, and the predicted conditional probability for each of the predicted values for the confounder variable.

19 . The non-transitory computer storage media of claim 18 , wherein determining a predicted probability for each of the predicted values of the confounder variable comprises:

processing a confounder prediction input generated from the scene data and the intervention data using a confounder prediction model, wherein the confounder prediction model is configured to receive the confounder prediction input and to process the confounder prediction input to generate a confounder distribution over a plurality of possible values for the confounder variable, wherein the confounder distribution comprises a predicted probability value for each of the possible values of the confounder variable.

20 . The non-transitory computer storage media of claim 19 , further comprising:

processing the scene data using an encoder neural network to generate an encoded representation of the scene data, wherein an input to the conditional behavior prediction model and the confounder prediction input are a same input that comprises the encoded representation and the intervention data.