Exploiting hierarchical structure learning with hyperbolic distance to enhance open world object detection
A method of performing open world object detection includes receiving object data, that includes embeddings data corresponding to a plurality of embeddings for known objects in a first input image, projecting the embeddings into a hyperbolic embedding space that includes embeddings in a plurality of categories of objects each including one or more classes of objects, regularizing the projected embeddings within the hyperbolic embedding space by moving each of the projected embeddings closer to embeddings in a same category of the plurality of categories and further away from embeddings in different categories of the plurality of categories, receiving an unmatched query corresponding to an object in a second input image, and generating, based on the hyperbolic embedding space including the regularized embeddings, an output signal that indicates whether the object in the second input image corresponds to an unknown object in one of the classes of objects.
1 . A method of performing open world object detection, the method comprising, using one or more processors:
receiving object data, wherein the object data includes embeddings data corresponding to embeddings for known objects in a first input image;
projecting the embeddings into a hyperbolic embedding space, wherein the hyperbolic embedding space includes embeddings in a plurality of categories of objects, and wherein each of the plurality of categories of objects includes one or more classes of objects;
regularizing the projected embeddings within the hyperbolic embedding space, wherein regularizing the projected embeddings within the hyperbolic embedding space includes determining respective hyperbolic averages of embeddings in each of the classes of objects, determining a threshold distance based on the hyperbolic averages, and moving each of the projected embeddings (i) closer to embeddings in a same category of the plurality of categories and (ii) further away from embeddings in different categories of the plurality of categories;
receiving an unmatched query corresponding to an object in a second input image; and
generating, in response to a determination that the unmatched query is less than the threshold distance from at least one of the hyperbolic averages, an output signal that indicates that the object in the second input image corresponds to an unknown object; and
controlling an operation of a computer-controlled machine in response to the output signal.
2 . The method of claim 1 , wherein the object data includes a set of bounding boxes and class labels for the known objects in the first input image.
3 . The method of claim 1 , wherein the first input image corresponds to a training image.
4 . The method of claim 1 , further comprising determining a hyperbolic contrastive loss corresponding to the hyperbolic embedding space, wherein regularizing the projected embeddings includes regularizing the projected embeddings based on the hyperbolic contrastive loss.
5 . A computing device configured to perform open world object detection, the computing device including a processing device configured to execute instructions stored in memory to:
receive object data, wherein the object data includes embeddings data corresponding to embeddings for known objects in a first input image;
project the embeddings into a hyperbolic embedding space, wherein the hyperbolic embedding space includes embeddings in a plurality of categories of objects, and wherein each of the plurality of categories of objects includes one or more classes of objects;
regularize the projected embeddings within the hyperbolic embedding space, wherein regularizing the projected embeddings within the hyperbolic embedding space includes determining respective hyperbolic averages of embeddings in each of the classes of objects determining a threshold distance based on the hyperbolic averages, and moving each of the projected embeddings (i) closer to embeddings in a same category of the plurality of categories and (ii) further away from embeddings in different categories of the plurality of categories;
receive an unmatched query corresponding to an object in a second input image; and
generate, in response to a determination that the unmatched query is less than the threshold distance from at least one of the hyperbolic averages, an output signal that indicates that the object in the second input image corresponds to an unknown object; and
control an operation of a computer-controlled machine in response to the output signal.
6 . The computing device of claim 5 , wherein the object data includes a set of bounding boxes and class labels for the known objects in the first input image.
7 . The computing device of claim 5 , wherein the first input image corresponds to a training image.
8 . The computing device of claim 5 , the processing device further configured to execute instructions stored in memory to generate a hyperbolic contrastive loss corresponding to the hyperbolic embedding space, wherein regularizing the projected embeddings includes regularizing the projected embeddings based on the hyperbolic contrastive loss.
9 . A computer-controlled machine, comprising:
at least one sensor configured to generate a first input image and a second input image;
a control system configured to
receive object data, wherein the object data includes embeddings data corresponding to embeddings for known objects in a first input image,
project the embeddings into a hyperbolic embedding space, wherein the hyperbolic embedding space includes embeddings in a plurality of categories of objects, and wherein each of the plurality of categories of objects includes one or more classes of objects,
regularize the projected embeddings within the hyperbolic embedding space, wherein regularizing the projected embeddings within the hyperbolic embedding space includes determining respective hyperbolic averages of embeddings in each of the classes of objects, determining a threshold distance based on the hyperbolic averages, and moving each of the projected embeddings (i) closer to embeddings in a same category of the plurality of categories and (ii) further away from embeddings in different categories of the plurality of categories,
receive an unmatched query corresponding to an object in a second input image, and
generate, in response to a determination that the unmatched query is less than the threshold distance from at least one of the hyperbolic averages, an output signal that indicates that the object in the second input image corresponds to an unknown object; and
an actuator configured to control an operation of the computer-controlled machine in response to the output signal.
10 . The computer-controlled machine of claim 9 , further comprising memory that stores data corresponding to the hyperbolic embedding space.
11 . The computer-controlled machine of claim 9 , wherein the control system is further configured to generate a hyperbolic contrastive loss corresponding to the hyperbolic embedding space, wherein regularizing the projected embeddings includes regularizing the projected embeddings based on the hyperbolic contrastive loss.