Semantic and instance segmentation

1. A computer-implemented method of detecting objects of interest comprising:

receiving, by a computing system comprising one or more computing devices, sensor data from one or more sensors configured to generate sensor data associated with an environment;

inputting, by the computing system, the sensor data to a machine-learned model configured to generate a class prediction and an instance prediction for each of a plurality of portions of the sensor data, the instance prediction for each of the plurality of portions of sensor data including an energy value based on a distance to at least one object boundary, the machine-learned model trained to generate a common energy value to represent the at least one object boundary;

generating, by the computing system as outputs of the machine-learned model, an instance prediction and a class prediction corresponding to each of the plurality of portions of the sensor data;

generating, by the computing system, one or more object segments based at least in part on the instance predictions and the class predictions; and

training the machine-learned model using a ground truth modified watershed transform as a target, wherein the target includes for the ground truth modified watershed transform the common energy value for separating a plurality of bins, wherein each bin corresponds to a different instance prediction.

2. The computer-implemented method of claim 1 , further comprising:

generating, by the computing system, an instance label for each of the plurality of portions of the sensor data based on a corresponding class prediction and instance prediction.

3. The computer-implemented method of claim 2 , wherein:

the machine-learned model includes a neural network that uses feature sharing on the sensor data at a first portion of the neural network associated with both class prediction and instance prediction;

the neural network includes a second portion that receives an output of the first portion of the neural network and determines the class prediction for each of the plurality of portions of the sensor data independently of the instance prediction; and

the neural network includes a third portion that receives the output of the first portion of the neural network and determines the instance prediction for each of the plurality of portions of the sensor data independently of the class prediction.

4. The computer-implemented method of claim 3 , wherein:

the first portion of the neural network includes a plurality of shared first layers;

the second portion of the neural network includes a plurality of unshared second layers configured to generate the class prediction for each of the plurality of portions of the sensor data; and

the third portion of the neural network includes a plurality of unshared third layers configured to generate the instance prediction for each of the plurality of portions of the sensor data.

5. The computer-implemented method of 2 , wherein the plurality of portions of the sensor data includes a first portion and generating an instance label for the first portion comprises:

determining, by the computing system, that the first portion is associated with a first class prediction;

in response to determining that the first portion is associated with the first class prediction, determining, by the computing system, that an energy value of the first portion satisfies a threshold; and

in response to determining that the energy value of the first portion satisfies the threshold, assigning, by the computing system, to the first portion a first instance label.

6. The computer-implemented method of claim 5 , further comprising:

determining, by the computing system, that each of one or more additional portions of the plurality of portions of sensor data is associated with the first class prediction and has an energy value that satisfies the threshold;

determining, by the computing system, that each of the one or more additional portions is within an object boundary corresponding to portions of the sensor data having energy values that do not satisfy the threshold; and

in response to determining that each of the one or more additional portions is within the object boundary, assigning, by the computing system, to each of the one or more additional portions the first instance label.

7. The computer-implemented method of claim 6 , wherein:

the common energy value for the at least one object boundary does not satisfy the threshold.

8. The computer-implemented method of claim 1 , further comprising:

providing, by the computing system, the one or more object segments to an object classification and tracking application; and

generating, by the computing system, one or more vehicle control signals for an autonomous vehicle based at least in part on an output of the object classification and tracking application.

9. The computer-implemented method of claim 1 , wherein:

the sensor data is image data; and

the machine-learned model comprises a convolutional neural network.

10. A computing system, comprising:

one or more processors;

a machine-learned model configured to receive sensor data representing an environment and in response to the sensor data to output an energy value in association with each of a plurality of portions of the sensor data, the machine-learned model trained to provide a common energy range to represent boundaries between objects in the environment; and

one or more non-transitory computer-readable media that store instructions, that when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:

providing, as input to the machine-learned model, sensor data representing the environment;

receiving, as output of the machine-learned model for each of a plurality of portions of the sensor data, a class prediction and an energy value, wherein the energy value is based on a distance of each portion relative to one or more object boundaries; and

generating an instance label for each of the plurality of portions of sensor data based on a corresponding energy value and class prediction, wherein generating an instance label for each of the plurality of portions of the sensor data comprises:

determining whether each of the plurality of portions of the sensor data is associated with a first class prediction;

for portions of the sensor data that are associated with the first class prediction, identifying one or more object boundaries based on sensor data that corresponds to an energy value that does not satisfy a threshold; and

for portions of the sensor data that are associated with the first class prediction and are within a common object boundary, generating a common instance label.

11. The computing system of claim 10 , wherein:

the machine-learned model is configured to output an energy map including the energy value for each of the plurality of portions of the sensor data; and

the energy map represents a plurality of objects in the environment.

12. The computing system of claim 10 , wherein:

the sensor data is image data; and

the machine-learned model comprises a convolutional neural network.

13. An autonomous vehicle, comprising:

a sensor system configured to generate sensor data of an environment external to the autonomous vehicle; and

a vehicle computing system, comprising:

one or more processors; and

one or more non-transitory computer-readable media that store instructions, that when executed by the one or more processors, cause the vehicle computing system to perform operations, the operations comprising:

providing sensor data from the sensor system as input to a machine-learned neural network including one or more shared layers configured to jointly determine instance information and classification information for each of a plurality of portions of the sensor data;

receiving, as a first output of one or more unshared first layers of the machine-learned neural network, an instance value for each of the plurality of portions of the sensor data based on the instance information from the one or more shared layers;

receiving, as a second output of one or more unshared second layers of the machine-learned neural network, a classification for each of the plurality of portions of the sensor data based on the classification information from the one or more shared layers; and

generating an instance label for each of the plurality of portions of the sensor data based on a corresponding classification and instance value from the machine-learned neural network, the instance value for each of the plurality of portions of the sensor data comprises an energy value based on a distance from each portion to a nearest object boundary, wherein the plurality of portions includes a first portion and generating an instance label for the first portion comprises:

determining if the first portion is associated with a first classification;

in response to determining that the first portion is associated with the first classification, determining if an energy value of the first portion satisfies a threshold; and

in response to determining that the energy value of the first portion satisfies the threshold, assigning to the first portion a first instance label.

14. The autonomous vehicle of claim 13 , wherein the operations further comprise:

determining whether each of one or more additional portions of the plurality of portions is associated with the first classification and has an energy value that satisfies the threshold; and

in response to determining that each of the one or more additional portions is associated with the first classification and has an energy value that satisfies the threshold, determining whether each of the one or more additional portions is within an object boundary corresponding to portions of the sensor data having energy values that do not satisfy the threshold.

15. The autonomous vehicle of claim 13 , wherein the operations further comprise:

training the machine-learned neural network using a ground truth modified watershed transform as a target;

wherein the target includes for the ground truth modified watershed transform a single energy value that separates a plurality of bins, wherein each bin corresponds to a different instance label.

16. A computer-implemented method of detecting objects of interest comprising:

receiving, by a computing system comprising one or more computing devices, sensor data from one or more sensors configured to generate sensor data associated with an environment;

inputting, by the computing system, the sensor data to a machine-learned model configured to generate a class prediction and an instance prediction for each of a plurality of portions of the sensor data, the instance prediction for each of the plurality of portions of sensor data including an energy value based on a distance to at least one object boundary, the machine-learned model trained to generate a common energy value to represent the at least one object boundary, wherein the machine-learned model includes a neural network that uses feature sharing on the sensor data at a first portion of the neural network associated with both class prediction and instance prediction, the neural network includes a second portion that receives an output of the first portion of the neural network and determines the class prediction for each of the plurality of portions of the sensor data independently of the instance prediction, the neural network includes a third portion that receives the output of the first portion of the neural network and determines the instance prediction for each of the plurality of portions of the sensor data independently of the class prediction;

generating, by the computing system as outputs of the machine-learned model, an instance prediction and a class prediction corresponding to each of the plurality of portions of the sensor data;

generating, by the computing system, one or more object segments based at least in part on the instance predictions and the class predictions; and

generating, by the computing system, an instance label for each of the plurality of portions of the sensor data based on a corresponding class prediction and instance prediction.