Systems and methods for identifying unknown instances
Systems and methods of the present disclosure provide an improved approach for open-set instance segmentation by identifying both known and unknown instances in an environment. For example, a method can include receiving sensor point cloud input data including a plurality of three-dimensional points. The method can include determining a feature embedding and at least one of an instance embedding, class embedding, and/or background embedding for each of the plurality of three-dimensional points. The method can include determining a first subset of points associated with one or more known instances within the environment based on the class embedding and the background embedding associated with each point in the plurality of points. The method can include determining a second subset of points associated with one or more unknown instances within the environment based on the first subset of points. The method can include segmenting the input data into known and unknown instances.
1. A method comprising:
providing a plurality of sensor cloud data points to a machine-learned instance detection model;
processing the plurality of sensor cloud data points with a first branch of the machine-learned instance detection model to detect one or more known instances within an environment, wherein the known instances comprise instances corresponding to one or more known classes of objects;
processing the plurality of sensor cloud data points with a second branch of the machine-learned instance detection model to detect one or more unknown instances within the environment, wherein the unknown instances comprise instances corresponding to one or more category-agnostic groupings; and
controlling motion of an autonomous platform based, at least in part, on the one or more known instances and the one or more unknown instances.
2. The method of claim 1 , wherein the first branch of the machine-learned instance detection model comprises a scoring head configured to determine a plurality of scores for the plurality of sensor cloud data points, wherein a respective score is indicative of a likelihood that a respective sensor cloud data point is associated with a known class in a closed set of the one or more known classes of objects.
3. The method of claim 1 , wherein the second branch of the machine-learned instance detection model comprises an embedding head configured to determine a plurality of point embeddings for the plurality of sensor cloud data points.
4. The method of claim 3 , wherein the plurality of point embeddings for the plurality of sensor cloud data points respectively comprise at least one of a class embedding, an instance embedding, or a background embedding.
5. The method of claim 4 , wherein:
the machine-learned instance detection model comprises a convolutional neural network; and
the second branch of the machine-learned instance detection model comprises a first convolutional layer configured to output the instance embedding, a second convolutional layer configured to output the class embedding, and a third convolutional layer configured to output the background embedding.
6. The method of claim 1 , further comprising:
processing the plurality of sensor cloud data points with a machine-learned feature embedding model to generate a plurality of feature embeddings for the plurality of sensor cloud data points; and
providing the plurality of sensor cloud data points and the plurality of feature embeddings to the first branch of the machine-learned instance detection model and to the second branch of the machine-learned instance detection model.
7. The method of claim 1 , wherein processing the plurality of sensor cloud data points with a second branch of the machine-learned instance detection model to detect one or more unknown instances within the environment comprises:
clustering unknown points of the plurality of sensor cloud data points into the one or more unknown instances; and
assigning an unknown semantic class label to the one or more unknown instances.
8. The method of claim 1 , wherein:
the first branch of the machine-learned instance detection model determines an anchor-point score for respective points in the plurality of sensor cloud data points; and
the one or more known instances within the environment are determined based on the anchor-point score for the respective points.
9. The method of claim 8 , wherein the anchor-point score is indicative of a likelihood of an anchor presence at a specific point of the respective points for a known class in the one or more known classes of objects.
10. The method of claim 1 , wherein the first branch and the second branch of the machine-learned instance detection model have been jointly trained end-to-end via backpropagation.
11. An autonomous vehicle (AV) control system comprising:
one or more processors; and
one or more tangible non-transitory computer readable media storing computer-readable instructions that are executable by the one or more processors to cause the one or more processors to perform operations, the operations comprising:
providing a plurality of sensor cloud data points to a machine-learned instance detection model;
processing the plurality of sensor cloud data points with a first branch of the machine-learned instance detection model to detect one or more known instances within an environment, wherein the known instances comprise instances corresponding to one or more known classes of objects;
processing the plurality of sensor cloud data points with a second branch of the machine-learned instance detection model to detect one or more unknown instances within the environment, wherein the unknown instances comprise instances corresponding to one or more category-agnostic groupings; and
controlling motion of an autonomous platform based, at least in part, on the one or more known instances and the one or more unknown instances.
12. The AV control system of claim 11 , wherein the first branch of the machine-learned instance detection model comprises a scoring head configured to determine a plurality of scores for the plurality of sensor cloud data points, wherein a respective score is indicative of a likelihood that a respective sensor cloud data point is associated with a known class in a closed set of the one or more known classes of objects.
13. The AV control system of claim 11 , wherein the second branch of the machine-learned instance detection model comprises an embedding head configured to determine a plurality of point embeddings for the plurality of sensor cloud data points.
14. The AV control system of claim 13 , wherein the plurality of point embeddings for the plurality of sensor cloud data points respectively comprise at least one of a class embedding, an instance embedding, or a background embedding.
15. The AV control system of claim 14 , wherein:
the machine-learned instance detection model comprises a convolutional neural network; and
the second branch of the machine-learned instance detection model comprises a first convolutional layer configured to output the instance embedding, a second convolutional layer configured to output the class embedding, and a third convolutional layer configured to output the background embedding.
16. The AV control system of claim 11 , the operations further comprising:
processing the plurality of sensor cloud data points with a machine-learned feature embedding model to generate a plurality of feature embeddings for the plurality of sensor cloud data points; and
providing the plurality of sensor cloud data points and the plurality of feature embeddings to the first branch of the machine-learned instance detection model and to the second branch of the machine-learned instance detection model.
17. The AV control system of claim 11 , wherein processing the plurality of sensor cloud data points with a second branch of the machine-learned instance detection model to detect one or more unknown instances within the environment comprises:
clustering unknown points of the plurality of sensor cloud data points into the one or more unknown instances; and
assigning an unknown semantic class label to the one or more unknown instances.
18. The AV control system of claim 11 , wherein:
the first branch of the machine-learned instance detection model determines an anchor-point score for respective points in the plurality of sensor cloud data points; and
the one or more known instances within the environment are determined based on the anchor-point score for the respective points.
19. The AV control system of claim 18 , wherein the anchor-point score is indicative of a likelihood of an anchor presence at a specific point of the respective points for a known class in the one or more known classes of objects.
20. An autonomous vehicle comprising:
one or more processors; and
one or more tangible non-transitory computer readable media storing computer-readable instructions that are executable by the one or more processors to cause the one or more processors to perform operations, the operations comprising:
providing a plurality of sensor cloud data points to a machine-learned instance detection model;
processing the plurality of sensor cloud data points with a first branch of the machine-learned instance detection model to detect one or more known instances within an environment, wherein the known instances comprise instances corresponding to one or more known classes of objects;
processing the plurality of sensor cloud data points with a second branch of the machine-learned instance detection model to detect one or more unknown instances within the environment, wherein the unknown instances comprise instances corresponding to one or more category-agnostic groupings; and
controlling motion of an autonomous platform based, at least in part, on the one or more known instances and the one or more unknown instances.