Co-learning object and relationship detection with density aware loss
An object detection model and relationship prediction model are jointly trained with parameters that may be updated through a joint backbone. The offset detection model predicts object locations based on keypoint detection, such as a heatmap local peak, enabling disambiguation of objects. The relationship prediction model may predict a relationship between detected objects and be trained with a joint loss with the object detection model. The loss may include terms for object connectedness and model confidence, enabling training to focus first on highly-connected objects and later on lower-confidence items.
1 . A system for image processing with object relationship detection, comprising:
a processor that executes instructions; and
a non-transitory computer-readable medium having instructions executable by the processor for:
determining a set of objects in an image based on an object detection model applied to the image, each object in the set of objects having a predicted object class;
for each object in the set of objects, determining a set of object relationship features based on a portion of a relation feature map of the image and the predicted object class, the relation feature map determined from a set of backbone layers shared with the object detection model; and
for one or more pairs of objects in the set of objects, predicting a relationship class for the pair of objects with a relationship prediction model based on the set of object relationship features of the respective objects.
2 . The system of claim 1 , wherein the object detection model identifies objects based on a central keypoint of the object.
3 . The system of claim 1 , wherein the object detection model determines an object class heatmap for an image based on a visual feature map of the image and determines the set of objects based on local peaks of each object class in the object class heatmap.
4 . The system of claim 3 , wherein the object detection model further determines an offset matrix and a size matrix based on the visual feature map and each object in the set of objects has a bounding box with coordinates determined based on the corresponding position of the local peaks in the offset matrix and the size matrix.
5 . The system of claim 1 , wherein the object detection model determines objects based on a visual feature map and the relation feature map is based on the visual feature map.
6 . The system of claim 1 , wherein for the one or more pairs of objects, the relationship prediction model predicts the relationship class with a direction between the objects.
7 . The system of claim 1 , wherein the instructions are further executable for training the object detection model features jointly with the relationship prediction model.
8 . The system of claim 7 , wherein parameters of the object detection model and relationship prediction model are continuously differentiable through joint backbone layers shared by the object detection model and the relationship prediction model.
9 . The system of claim 7 , wherein the relationship prediction model is trained with a loss function for the relationship prediction model that includes a component for the predicted relationship class, a predicted subject object class and a predicted object class.
10 . The system of claim 7 , wherein a loss function for the relationship prediction model includes a density and confidence-based loss that increases the weight for objects having a relatively high number of relationships to other objects and decreases the weight for relationship predictions having a relatively high confidence of predicted relationship class.
11 . A method for image processing with object relationship detection, comprising:
determining a set of objects in an image based on an object detection model applied to the image, each object in the set of objects having a predicted object class;
for each object in the set of objects, determining a set of object relationship features based on a portion of a relation feature map of the image and the predicted object class, the relation feature map determined from a set of backbone layers shared with the object detection model; and
for one or more pairs of objects in the set of objects, predicting a relationship class for the pair of objects with a relationship prediction model based on the set of object relationship features of the respective objects.
12 . The method of claim 11 , wherein the object detection model identifies objects based on a central keypoint of the object.
13 . The method of claim 11 , wherein the object detection model determines an object class heatmap for an image based on a visual feature map of the image and determines the set of objects based on local peaks of each object class in the object class heatmap.
14 . The method of claim 13 , wherein the object detection model further determines an offset matrix and a size matrix based on the visual feature map and each object in the set of objects has a bounding box with coordinates determined based on the corresponding position of the local peaks in the offset matrix and the size matrix.
15 . The method of claim 11 , wherein the object detection model determines objects based on a visual feature map and the relation feature map is based on the visual feature map.
16 . The method of claim 11 , wherein for the one or more pairs of objects, the relationship prediction model predicts the relationship class with a direction between the objects.
17 . The method of claim 11 , the instructions being further for training the object detection model features jointly with the relationship prediction model.
18 . The method of claim 17 , wherein parameters of the object detection model and relationship prediction model are continuously differentiable through joint backbone layers shared by the object detection model and the relationship prediction model.
19 . The method of claim 17 , wherein the relationship prediction model is trained with a loss function for the relationship prediction model that includes a component for the predicted relationship class, a predicted subject object class and a predicted object class.
20 . The method of claim 17 , wherein a loss function for the relationship prediction model includes a density and confidence-based loss that increases the weight for objects having a relatively high number of relationships to other objects and decreases the weight for relationship predictions having a relatively high confidence of predicted relationship class.