Joint detection and grouping of road objects using machine learning

1 . A computer-implemented method comprising:

obtaining an image representing a plurality of road objects belonging to a particular class;

generating, using an encoder model and based on the image, a plurality of feature maps that represent visual features of the image;

generating, by processing the plurality of feature maps using a keypoint machine learning model, a keypoint map comprising a first plurality of pixels, wherein a corresponding value of each respective pixel of the first plurality of pixels indicates a likelihood that the respective pixel represents a corresponding road object of the plurality of road objects;

determining, based on the keypoint map and for each respective road object of the plurality of road objects, a corresponding location of the respective road object within the image;

generating, by processing the plurality of feature maps using a tag machine learning model, a tag map comprising a second plurality of pixels, wherein each respective pixel of the second plurality of pixels comprises a respective tag value that indicates a group to which an object represented by the respective pixel belongs;

determining, using the tag map and for each respective road object of the plurality of road objects, a corresponding tag value associated with the respective road object based on the corresponding location of the respective road object;

determining one or more groups of the plurality of road objects based on the tag value of each respective road object of the plurality of road objects by grouping road objects based on a similarity of the corresponding tag values of the plurality of road objects, wherein each respective group of the one or more groups comprises two or more corresponding road objects;

identifying, for each respective group of the one or more groups, a corresponding road condition based on (i) the corresponding locations of the two or more corresponding road objects in the respective group and (ii) the particular class of the plurality of road objects; and

controlling operation of a vehicle based on the corresponding road condition of at least one group of the one or more groups.

2 . The computer-implemented method of claim 1 , wherein determining the corresponding location at which each respective road object has been detected within the image comprises:

identifying, within the keypoint map, two or more pixels associated with corresponding values that exceed a threshold value, wherein respective positions of the two or more pixels within the keypoint map represent the corresponding locations of the plurality of road objects.

3 . The computer-implemented method of claim 1 , wherein the likelihood that the respective pixel represents a corresponding road object indicates a likelihood that the respective pixel represents a predetermined portion of the corresponding road object, and wherein the predetermined portion of the corresponding road object comprises one or more of: (i) a point along a vertical center line of the corresponding road object, (ii) a point along a horizontal center line of the corresponding road object, (iii) a point on a bounding box of the corresponding road object, or (iv) a center point of the bounding box of the corresponding road object.

4 . The computer-implemented method of claim 1 , wherein determining the corresponding location of the respective road object within the image comprises:

determining, using an offset machine learning model and based on the plurality of feature maps, an offset map comprising a third plurality of pixels, wherein each respective pixel of the third plurality of pixels that represents a corresponding offset between (i) a location within the keypoint map of a corresponding pixel of the first plurality of pixels and (ii) a location within the image of a feature represented by the corresponding pixel of the first plurality of pixels; and

determining, for each respective road object of the plurality of road objects, the corresponding location of the respective road object within the image by adding (i) the corresponding offset to (ii) a corresponding location of the respective road object within the keypoint map.

5 . The computer-implemented method of claim 1 , wherein

the respective tag value of each respective pixel of the second plurality of pixels comprises a corresponding tag vector comprising a plurality of values and configured to indicate the group to which the object represented by the respective pixel belongs.

6 . The computer-implemented method of claim 5 , wherein determining the one or more groups comprises:

determining, for each respective road object of the plurality of road objects, one or more corresponding distance metrics between (i) the corresponding tag vector associated with the respective road object and (ii) one or more tag vectors associated with one or more other road objects of the plurality of road objects; and

partitioning the plurality of road objects into the one or more groups based on the one or more corresponding distance metrics of each respective road object of the plurality of road objects.

7 . The computer-implemented method of claim 1 , wherein

the respective tag value of each respective pixel of the second plurality of pixels comprises a corresponding scalar value associated with the respective pixel and configured to indicate the group to which the object represented by the respective pixel belongs.

8 . The computer-implemented method of claim 7 , wherein determining the one or more groups comprises:

determining, for each respective road object of the plurality of road objects, the corresponding scalar value associated with the respective road object; and

partitioning the plurality of road objects into the one or more groups based on the corresponding scalar value of each respective road object of the plurality of road objects.

9 . The computer-implemented method of claim 1 , wherein the tag machine learning model comprises a semantic embedding model and a relation reasoning machine learning (ML) model, and wherein determining the corresponding tag value comprises:

generating, using the semantic embedding model, based on the plurality of feature maps, and for each respective road object of the plurality of road objects, a corresponding semantic embedding vector;

generating, for each respective road object of the plurality of road objects, a corresponding position-aware semantic embedding vector by combining (i) the corresponding semantic embedding vector of the respective road object with (ii) a positional embedding vector of the respective road object; and

generating, using the relation reasoning ML model and for each respective road object of the plurality of road objects, the corresponding tag value based on the corresponding position-aware semantic embedding vector.

10 . The computer-implemented method of claim 9 , wherein the relation reasoning ML model comprises a self-attention model configured to determine, for each respective road object of the plurality of road objects, a corresponding plurality of attention scores between (i) the corresponding position-aware embedding vector of the respective road object and (ii) the corresponding position-aware embedding vectors of other road objects of the plurality of road objects, wherein the corresponding tag value of the respective road object is based on the corresponding plurality of attention scores.

11 . The computer-implemented method of claim 9 , wherein the plurality of road objects are represented as nodes of a graph, and wherein the relation reasoning ML model comprises a graph neural network model configured to determine, for each respective road object of the plurality of road objects, the corresponding tag value based on one or more corresponding position-aware embedding vectors of one or more other road objects of the plurality of road objects, wherein each of the one or more other road objects is linked to the respective road object by a corresponding edge of the graph.

12 . The computer-implemented method of claim 1 , wherein the plurality of road objects are configured to indicate construction zones, wherein the corresponding road condition represents a corresponding construction zone in a region of environment indicated by the corresponding locations of the two or more corresponding road objects in the respective group.

13 . The computer-implemented method of claim 1 , wherein the keypoint machine learning model has been trained by a training process comprising:

obtaining a training sample comprising (i) a training image representing a plurality of training road objects belonging to the particular class and (ii), for each respective training road object of the plurality of training road objects, a corresponding ground-truth location of the respective training road object within the training image;

generating, using the encoder model and based on the training image, a plurality of training feature maps that represent visual features of the training image;

determining, using the keypoint machine learning model, based on the plurality of training feature maps, and for each respective training road object of the plurality of training road objects, a corresponding training location at which the respective training road object has been detected within the training image;

determining a focal loss value by comparing, for each respective training road object of the plurality of training road objects and using a focal loss function, the corresponding ground-truth location of the respective training road object to the corresponding training location of the respective training road object; and

adjusting one or more parameters of the keypoint machine learning model based on the focal loss value.

14 . The computer-implemented method of claim 1 , wherein the tag machine learning model has been trained by a training process comprising:

obtaining a training sample comprising (i) a training image representing a plurality of training road objects belonging to the particular class and (ii) one or more ground-truth groups of the plurality of road objects;

generating, using the encoder model and based on the training image, a plurality of training feature maps that represent visual features of the training image;

determining, using the tag machine learning model, based on the plurality of training feature maps, and for each respective training road object of the plurality of training road objects, a corresponding training tag value associated with the respective training road object;

determining one or more training groups of the plurality of training road objects based on the corresponding training tag value of each respective training road object of the plurality of training road objects;

determining a grouping loss value by comparing, using a grouping loss function, the one or more ground-truth groups to the one or more training groups, wherein the grouping loss function is configured to, for each respective training road object of the plurality of training road objects, cause the corresponding training tag value to (i) converge to first corresponding tag values of a first set of training road objects within a ground-truth group of the respective training road object and (ii) diverge from second corresponding tag values of a second set of training road objects outside of the ground-truth group; and

adjusting one or more parameters of the tag machine learning model based on the grouping loss value.

15 . The computer-implemented method of claim 1 , wherein identifying the corresponding road condition comprises:

identifying, for each respective group of the one or more groups and based on (i) the corresponding locations of the two or more corresponding road objects in the respective group and (ii) the particular class of the plurality of road objects, a corresponding environment region in which the corresponding road condition is present, wherein the operation of the vehicle is controlled based on the corresponding environment region of the at least one group of the one or more groups.

16 . The computer-implemented method of claim 1 , further comprising:

generating, based on the image, an annotated image that represents the corresponding road condition indicated by each respective group of the one or more groups.

17 . A system comprising:

a camera;

a processor; and

a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations comprising:

obtaining, from the camera, an image representing a plurality of road objects belonging to a particular class;

generating, using an encoder model and based on the image, a plurality of feature maps that represent visual features of the image;

generating, by processing the plurality of feature maps using a keypoint machine learning model, a keypoint map comprising a first plurality of pixels, wherein a corresponding value of each respective pixel of the first plurality of pixels indicates a likelihood that the respective pixel represents a corresponding road object of the plurality of road objects;

determining, based on the keypoint map and for each respective road object of the plurality of road objects, a corresponding location of the respective road object within the image;

generating, by processing the plurality of feature maps using a tag machine learning model, a tag map comprising a second plurality of pixels, wherein each respective pixel of the second plurality of pixels comprises a respective tag value that indicates a group to which an object represented by the respective pixel belongs;

determining, using the tag map and for each respective road object of the plurality of road objects, a corresponding tag value associated with the respective road object based on the corresponding location of the respective road object;

determining one or more groups of the plurality of road objects based on the tag value of each respective road object of the plurality of road objects by grouping road objects based on a similarity of the corresponding tag values of the plurality of road objects, wherein each respective group of the one or more groups comprises two or more corresponding road objects;

identifying, for each respective group of the one or more groups, a corresponding road condition based on (i) the corresponding locations of the two or more corresponding road objects in the respective group and (ii) the particular class of the plurality of road objects; and

controlling operation of a vehicle based on the corresponding road condition of at least one group of the one or more groups.

18 . The system of claim 17 , wherein the tag machine learning model comprises a semantic embedding model and a relation reasoning machine learning (ML) model, and wherein determining the corresponding tag value comprises:

generating, using the semantic embedding model, based on the plurality of feature maps, and for each respective road object of the plurality of road objects, a corresponding semantic embedding vector;

generating, for each respective road object of the plurality of road objects, a corresponding position-aware semantic embedding vector by combining (i) the corresponding semantic embedding vector of the respective road object with (ii) a positional embedding vector of the respective road object; and

generating, using the relation reasoning ML model and for each respective road object of the plurality of road objects, the corresponding tag value based on the corresponding position-aware semantic embedding vector.

19 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations comprising:

obtaining an image representing a plurality of road objects belonging to a particular class;

generating, using an encoder model and based on the image, a plurality of feature maps that represent visual features of the image;

generating, by processing the plurality of feature maps using a keypoint machine learning model, a keypoint map comprising a first plurality of pixels, wherein a corresponding value of each respective pixel of the first plurality of pixels indicates a likelihood that the respective pixel represents a corresponding road object of the plurality of road objects;

determining, based on the keypoint map and for each respective road object of the plurality of road objects, a corresponding location of the respective road object within the image;

generating, by processing the plurality of features maps using a tag machine learning model, a tag map comprising a second plurality of pixels, wherein each respective pixel of the second plurality of pixels comprises a respective tag value that indicates a group to which an object represented by the respective pixel belongs;

determining, using the tag map and for each respective road object of the plurality of road objects, a corresponding tag value associated with the respective road object based on the corresponding location of the respective road object;

determining one or more groups of the plurality of road objects based on the tag value of each respective road object of the plurality of road objects by grouping road objects based on a similarity of the corresponding tag values of the plurality of road objects, wherein each respective group of the one or more groups comprises two or more corresponding road objects;

identifying, for each respective group of the one or more groups, a corresponding road condition based on (i) the corresponding locations of the two or more corresponding road objects in the respective group and (ii) the particular class of the plurality of road objects; and

controlling operation of a vehicle based on the corresponding road condition of at least one group of the one or more groups.

20 . The computer-implemented method of claim 1 , wherein the keypoint map and the tag map are generated non-serially such that generation of the keypoint map is independent of generation of the tag map and generation of the tag map is independent of generation of the keypoint map.