Resolving object detection outputs from multiple object detection pipelines

1 . A computer-implemented method, comprising:

generating, by a first object detection model of a perception system of an autonomous vehicle and based on a first portion of sensor data describing a first portion of an environment of the autonomous vehicle, a first object detection output that comprises a first predicted class for an object;

generating, by a second object detection model of the perception system and based on a second portion of sensor data describing a second portion of the environment, a second object detection output that comprises a predicted distribution over candidate classes for the object;

querying a data structure comprising one or more precomputed logit estimation tensors using the first predicted class;

retrieving a precomputed logit tensor associated with the first predicted class and the first object detection model, wherein the precomputed logit tensor indicates an estimated distribution over candidate classes;

adapting, by a prediction resolution model, at least a portion of the first object detection output into an adapted representation of the first object detection output;

providing the adapted representation in a shared prediction output space with the predicted distribution from the second object detection output; and

generating, by the prediction resolution model, a resolved object detection output based on the adapted representation and the predicted distribution, wherein the resolved object detection output indicates a second predicted class for the object.

2 . A computer-implemented method, comprising:

generating, by a first object detection model of a perception system of an autonomous vehicle and based on a first portion of sensor data describing a first portion of an environment of the autonomous vehicle, a first object detection output that comprises a first predicted class for an object;

generating, by a second object detection model of the perception system and based on a second portion of sensor data describing a second portion of the environment, a second object detection output that comprises a predicted distribution over candidate classes for the object;

adapting by a prediction resolution model, at least a portion of the first object detection output into an adapted representation of the first object detection output, wherein one or more output layers of the prediction resolution model are optimized using a global optimizer over a single batch of ground truth examples;

providing the adapted representation in a shared prediction output space with the predicted distribution from the second object detection output; and

generating, by the prediction resolution model, a resolved object detection output based on the adapted representation and the predicted distribution, wherein the resolved object detection output indicates a second predicted class for the object.

3 . A computer-implemented method, comprising:

generating, by a first object detection model of a perception system of an autonomous vehicle and based on a first portion of sensor data describing a first portion of an environment of the autonomous vehicle, a first object detection output that comprises a first predicted class for an object;

generating, by a second object detection model of the perception system and based on a second portion of sensor data describing a second portion of the environment, a second object detection output that comprises a predicted distribution over candidate classes for the object;

adapting, by a prediction resolution model, at least a portion of the first object detection output into an adapted representation of the first object detection output, wherein one or more output layers of the prediction resolution model are optimized using a non-uniformly downsampled batch of a dataset of ground truth examples, wherein a ratio of a number of a respective category of examples in the batch to a number of the respective category in the dataset is inversely correlated with an error rate associated with the respective category;

providing the adapted representation in a shared prediction output space with the predicted distribution from the second object detection output; and

generating, by the prediction resolution model, a resolved object detection output based on the adapted representation and the predicted distribution, wherein the resolved object detection output indicates a second predicted class for the object.

4 . A computer-implemented method, comprising:

generating, at a first time, by a first object detection model of a perception system of an autonomous vehicle and based on a first portion of sensor data describing a first portion of an environment of the autonomous vehicle, a first object detection output;

generating, at a second time, by a second object detection model of the perception system and based on a second portion of sensor data describing a second portion of the environment, a second object detection output; and

generating, by a prediction resolution model and based on the first object detection output and the second object detection output in a shared prediction output space, a resolved object detection output, wherein the prediction resolution model executes periodically to resolve conflicts between object detection outputs from the first object detection model and the second object detection model, the prediction resolution model discounting a contribution of the first object detection model based on the first time or discounting a contribution of the second object detection model based on the second time.

5 . The computer-implemented method of claim 4 , wherein:

the first object detection output comprises first update data for an object track stored by the perception system to track an object in the environment, wherein the first update data indicates a first update to the object track; and

the second object detection output comprises second update data for the object track, wherein the second update data indicates a second update to the object track.

6 . The computer-implemented method of claim 4 , wherein the first portion of sensor data comprises a modality of data not present in the second portion of sensor data.

7 . The computer-implemented method of claim 4 , wherein the first portion of sensor data comprises data describing a different field of view of the environment as compared to the second portion of sensor data.

8 . A computer-implemented method, comprising:

generating, at a first time, by a first object detection model of a perception system of an autonomous vehicle and based on a first portion of sensor data describing a first portion of an environment of the autonomous vehicle, a first object detection output;

generating, at a second time, by a second object detection model of the perception system and based on a second portion of sensor data describing a second portion of the environment, a second object detection output;

generating, by a prediction resolution model and based on the first object detection output and the second object detection output in a shared prediction output space, a resolved object detection output, wherein the prediction resolution model executes periodically to resolve conflicts between object detection outputs from the first object detection model and the second object detection model;

adapting, by the prediction resolution model, at least a portion of the first object detection output to obtain a representation of the first object detection output in the shared prediction output space; and

generating the resolved object detection output based on the representation of the first object detection output in the shared prediction output space.

9 . A computer-implemented method, comprising:

generating, at a first time, by a first object detection model of a perception system of an autonomous vehicle and based on a first portion of sensor data describing a first portion of an environment of the autonomous vehicle, a first object detection output;

generating, at a second time, by a second object detection model of the perception system and based on a second portion of sensor data describing a second portion of the environment, a second object detection output;

generating, by a prediction resolution model and based on the first object detection output and the second object detection output in a shared prediction output space, a resolved object detection output, wherein the prediction resolution model executes periodically to resolve conflicts between object detection outputs from the first object detection model and the second object detection model;

processing, by the prediction resolution model, a predicted class from the first object detection output;

processing, by the prediction resolution model, a predicted distribution over candidate classes from the second object detection output;

adapting, by the prediction resolution model, at least a portion of the first object detection output into an adapted representation of the first object detection output;

providing the adapted representation in a shared prediction output space with the predicted distribution from the second object detection output; and

generating, by the prediction resolution model, the resolved object detection output based on the adapted representation and the predicted distribution.

10 . The computer-implemented method of claim 4 , wherein one or more output layers of the prediction resolution model are optimized using a global optimizer over a single batch of ground truth examples.

11 . The computer-implemented method of claim 4 , wherein one or more output layers of the prediction resolution model are optimized using a non-uniformly downsampled batch of a dataset of ground truth examples, wherein a ratio of a number of a respective category of examples in the batch to a number of the respective category in the dataset is inversely correlated with an error rate associated with the respective category.

12 . A computing system comprising:

one or more processors; and

one or more non-transitory computer-readable media storing instructions that are executable by the one or more processors to cause the computing system to perform operations comprising:

generating, at a first time, by a first object detection model of a perception system of an autonomous vehicle and based on a first portion of sensor data describing a first portion of an environment of the autonomous vehicle, a first object detection output;

generating, at a second time, by a second object detection model of the perception system and based on a second portion of sensor data describing a second portion of the environment, a second object detection output; and

generating, by a prediction resolution model and based on the first object detection output and the second object detection output in a shared prediction output space, a resolved object detection output, wherein the prediction resolution model executes periodically to resolve conflicts between object detection outputs from the first object detection model and the second object detection model, the prediction resolution model discounting a contribution of the first object detection model based on the first time or discounting a contribution of the second object detection model based on the second time.

13 . The computing system of claim 12 , wherein:

the first object detection output comprises first update data for an object track stored by the perception system to track an object in the environment, wherein the first update data indicates a first update to the object track; and

the second object detection output comprises second update data for the object track, wherein the second update data indicates a second update to the object track.

14 . The computing system of claim 12 , wherein the first portion of sensor data comprises a modality of data not present in the second portion of sensor data.

15 . The computing system of claim 12 , wherein the first portion of sensor data comprises data describing a different field of view of the environment as compared to the second portion of sensor data.

16 . The computing system of claim 12 , the operations comprising:

adapting, by the prediction resolution model, at least a portion of the first object detection output to obtain a representation of the first object detection output in the shared prediction output space; and

generating the resolved object detection output based on the representation of the first object detection output in the shared prediction output space.

17 . The computing system of claim 12 , the operations comprising:

processing, by the prediction resolution model, a predicted class from the first object detection output;

processing, by the prediction resolution model, a predicted distribution over candidate classes from the second object detection output;

adapting, by the prediction resolution model, at least a portion of the first object detection output into an adapted representation of the first object detection output;

providing the adapted representation in a shared prediction output space with the predicted distribution from the second object detection output; and

generating, by the prediction resolution model, the resolved object detection output based on the adapted representation and the predicted distribution.

18 . The computing system of claim 12 , wherein one or more output layers of the prediction resolution model are optimized using a global optimizer over a single batch of ground truth examples.

19 . The computing system of claim 12 , wherein one or more output layers of the prediction resolution model are optimized using a non-uniformly downsampled batch of a dataset of ground truth examples, wherein a ratio of a number of a respective category of examples in the batch to a number of the respective category in the dataset is inversely correlated with an error rate associated with the respective category.

20 . One or more non-transitory computer-readable media storing instructions that are executable by one or more processors to cause a computing system to perform operations comprising:

generating, at a first time, by a first object detection model of a perception system of an autonomous vehicle and based on a first portion of sensor data describing a first portion of an environment of the autonomous vehicle, a first object detection output;

generating, at a second time, by a second object detection model of the perception system and based on a second portion of sensor data describing a second portion of the environment, a second object detection output; and

generating, by a prediction resolution model and based on the first object detection output and the second object detection output in a shared prediction output space, a resolved object detection output, wherein the prediction resolution model executes periodically to resolve conflicts between object detection outputs from the first object detection model and the second object detection model, the prediction resolution model discounting a contribution of the first object detection model based on the first time or discounting a contribution of the second object detection model based on the second time.

21 . The one or more non-transitory computer-readable media of claim 20 , wherein:

the first object detection output comprises first update data for an object track stored by the perception system to track an object in the environment, wherein the first update data indicates a first update to the object track; and

the second object detection output comprises second update data for the object track, wherein the second update data indicates a second update to the object track.

22 . The one or more non-transitory computer-readable media of claim 20 , wherein the first portion of sensor data comprises a modality of data not present in the second portion of sensor data.

23 . The one or more non-transitory computer-readable media of claim 20 , wherein the first portion of sensor data comprises data describing a different field of view of the environment as compared to the second portion of sensor data.