Trajectory determination incorporating remote operations

1 . A system comprising:

one or more processors; and

one or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause the system to perform operations comprising:

receiving context data representing a current state of an environment of a vehicle at a current time;

receiving object historical data representing a behavior of an object in the environment during a period before the current time;

receiving vehicle historical data representing a behavior of the vehicle in the environment during the period before the current time;

determining, for a future time, prediction data comprising one or more of a predicted vehicle state or a predicted object state;

determining, based at least in part on the context data, the object historical data, the vehicle historical data, and the prediction data, a first data associated with a predicted state of the environment at the future time;

receiving teleoperation data associated with a teleoperation command received by the vehicle at the current time, the teleoperation data associated with controlling the vehicle through the environment;

determining, based at least in part on the teleoperation data, second data comprising an embedding associated with the teleoperation data;

determining, based at least in part on processing the first data and the second data, a cost associated with a candidate trajectory; and

controlling the vehicle based at least in part on the cost.

2 . The system of claim 1 , wherein determining the cost comprises determining a combined embedding by concatenating the first data and the second data.

3 . The system of claim 1 , wherein:

the teleoperation command is associated with at least one of:

a vertical position shift of the vehicle, a horizontal position shift of the vehicle, a heading of the vehicle, a velocity of the vehicle, an acceleration of the vehicle, or a yaw rate of the vehicle.

4 . The system of claim 1 , wherein determining the prediction data comprises:

generating a tree structure associated with a plurality of actions for the vehicle to perform; and

determining, for a trace of the tree structure, the predicted state of the vehicle.

5 . The system of claim 1 , wherein the cost is a first cost associated with a first cost category, the operations further comprising:

determining, based at least in part on inputting the first data and the second data into a second portion of a machine learned model, a second cost, wherein controlling the vehicle is further based at least in part on the second cost.

6 . One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:

receiving prediction data representing a predicted state of an environment of a vehicle at a future time;

receiving teleoperation data representing a teleoperation command received by the vehicle;

determining, based at least in part on the teleoperation command, an embedding associated with the teleoperation data;

determining, based at least in part on the prediction data and the embedding, a cost associated with controlling the vehicle; and

controlling the vehicle based at least in part on the cost.

7 . The one or more non-transitory computer-readable media of claim 6 , the operations further comprising:

determining whether to ignore an impact of the teleoperation command on control of the vehicle; and

based at least in part on determining to ignore the impact of the teleoperation command on control of the vehicle, setting the teleoperation data to a default value.

8 . The one or more non-transitory computer-readable media of claim 6 , wherein:

the teleoperation data represents that the vehicle has received the teleoperation command, the teleoperation command comprises a sequence of actions, and determining the teleoperation data comprises:

providing the sequence of actions to a machine-learned model, and receiving, from the machine-learned model, the teleoperation data.

9 . The one or more non-transitory computer-readable media of claim 8 , wherein the machine-learned model comprises at least one of:

a recurrent neural network, or an attention-based transformer.

10 . The one or more non-transitory computer-readable media of claim 6 , wherein the operations further comprise:

receiving context data representing a current state of the environment at a current time;

receiving object historical data representing a behavior of an object in the environment during a period before the current time; and

receiving vehicle historical data representing a behavior of the vehicle in the environment during the period before the current time;

wherein determining the cost is further based at least in part on the context data, the object historical data, and the vehicle historical data.

11 . The one or more non-transitory computer-readable media of claim 10 , wherein receiving the context data comprises:

receiving a top-down representation of the environment at the current time;

providing the top-down representation to a machine-learned model, wherein the machine-learned model comprises a convolutional neural network; and

receiving the context data from the machine-learned model.

12 . The one or more non-transitory computer-readable media of claim 10 , wherein receiving the object historical data comprises:

receiving a sequence of actions associated with the object during the period;

providing the sequence of actions to a machine-learned model, wherein the machine-learned model comprises a recurrent neural network; and

receiving the object historical data from the machine-learned model.

13 . The one or more non-transitory computer-readable media of claim 6 , wherein the teleoperation command is associated with at least one of:

a vertical position shift associated with the vehicle, a horizontal position shift associated with the vehicle, a heading associated with the vehicle, a velocity associated with the vehicle, an acceleration associated with the vehicle, or a yaw rate associated with the vehicle.

14 . The one or more non-transitory computer-readable media of claim 6 , wherein determining the embedding comprises processing the teleoperation command using a machine-learned model.

15 . A method comprising:

receiving prediction data representing a predicted state of an environment of a vehicle at a future time;

receiving teleoperation data representing a teleoperation command received by the vehicle;

determining, based at least in part on the teleoperation command, an embedding associated with the teleoperation data;

determining, based at least in part on the prediction data and the embedding, a cost associated with controlling the vehicle; and

controlling the vehicle based at least in part on the cost.

16 . The method of claim 15 , further comprising:

determining whether to ignore an impact of the teleoperation command on control of the vehicle; and

based at least in part on determining to ignore the impact of the teleoperation command on control of the vehicle, setting the teleoperation data to a default value.

17 . The method of claim 15 , wherein:

the teleoperation data represents that the vehicle has received the teleoperation command, the teleoperation command comprises a sequence of actions, and determining the teleoperation data comprises:

providing the sequence of actions to a machine-learned model, and receiving, from the machine-learned model, the teleoperation data.

18 . The method of claim 17 , wherein the machine-learned model comprises at least one of:

a recurrent neural network, or an attention-based transformer.

19 . The method of claim 15 , further comprising:

receiving context data representing a current state of the environment at a current time;

receiving object historical data representing a behavior of an object in the environment during a period before the current time; and

receiving vehicle historical data representing a behavior of the vehicle in the environment during the period before the current time;

wherein determining the cost is further based at least in part on the context data, the object historical data, and the vehicle historical data.

20 . The method of claim 19 , wherein receiving the object historical data comprises:

receiving a sequence of actions associated with the object during the period;

providing the sequence of actions to a machine-learned model, wherein the machine-learned model comprises a recurrent neural network; and

receiving the object historical data from the machine-learned model.