Message passing network based object signature for object tracking
Disclosed are systems, apparatuses, processes, and computer-readable media for processing image data. For example, an apparatus can compute initial embeddings from a plurality of images. The apparatus can construct a graph comprising nodes representing the initial embeddings. The apparatus can further perform, based on the graph, a plurality of message passing steps successively to generate final embeddings. The apparatus can classify, using a classification engine, one or more objects in each of the plurality of images based on the final embeddings. The apparatus can further compute a classification loss based on the classifying of the one or more objects.
1 . A processor-implemented method of processing image data, comprising:
computing initial embeddings from a plurality of images, wherein the initial embeddings comprise baseline object signatures for objects represented in the plurality of images;
constructing a graph comprising nodes and edges, the nodes representing the initial embeddings of respective objects represented in the plurality of images, each edge having an edge weight based on similarity between the baseline object signatures of the respective objects;
performing based on the graph, a plurality of message passing steps to generate final embeddings;
classifying, using a classification engine, one or more objects in each of the plurality of images based on the final embeddings; and
adjusting an operation of a vehicle based on the classifying of the one or more objects.
2 . The processor-implemented method of claim 1 , wherein the initial embeddings are computed using a neural network.
3 . The processor-implemented method of claim 2 , further comprising tuning one or more weights of the neural network based on a classification loss computed based on the classifying of the one or more objects.
4 . The processor-implemented method of claim 1 , wherein the graph is constructed using a message passing neural network.
5 . The processor-implemented method of claim 4 , wherein the plurality of message passing steps are performed using the message passing neural network.
6 . The processor-implemented method of claim 1 , further comprising applying an attention score at each of the plurality of message passing steps for each of the nodes of the graph to weigh an importance of neighboring nodes.
7 . The processor-implemented method of claim 1 , wherein messages are passed between all of the nodes of the graph during each of the plurality of message passing steps.
8 . The processor-implemented method of claim 3 , wherein the classification loss is based on one of a triplet loss or a fast-approximated triplet (FAT) loss.
9 . The processor-implemented method of claim 1 , further comprising applying, by the classification engine, a cross-entropy loss to the final embeddings.
10 . The processor-implemented method of claim 1 , wherein the graph is a fully-connected graph.
11 . An apparatus for processing image data, comprising:
at least one memory; and
at least one processor coupled to the at least one memory and configured to:
compute initial embeddings from a plurality of images, wherein the initial embeddings comprise baseline object signatures for objects represented in the plurality of images;
construct a graph comprising nodes and edges, the nodes representing the initial embeddings of respective objects represented in the plurality of images, each edge having an edge weight based on similarity between the baseline object signatures of the respective objects;
perform, based on the graph, a plurality of message passing steps to generate final embeddings;
classify, using a classification engine, one or more objects in each of the plurality of images based on the final embeddings; and
adjust an operation of a vehicle based on the classifying of the one or more objects.
12 . The apparatus of claim 11 , wherein the at least one processor is configured to compute the initial embeddings using a neural network.
13 . The apparatus of claim 12 , wherein the at least one processor is configured to tune one or more weights of the neural network based on a classification loss computed based on the classifying of the one or more objects.
14 . The apparatus of claim 11 , wherein the at least one processor is configured to construct the graph using a message passing neural network.
15 . The apparatus of claim 14 , wherein the at least one processor is configured to perform the plurality of message passing steps using the message passing neural network.
16 . The apparatus of claim 11 , wherein the at least one processor is configured to apply an attention score at each of the plurality of message passing steps for each of the nodes of the graph to weigh an importance of neighboring nodes.
17 . The apparatus of claim 11 , wherein the at least one processor is configured to pass messages between all of the nodes of the graph during each of the plurality of message passing steps.
18 . The apparatus of claim 13 , wherein the classification loss is based on one of a triplet loss or a fast-approximated triplet (FAT) loss.
19 . The apparatus of claim 11 , wherein the at least one processor is configured to apply, using the classification engine, a cross-entropy loss to the final embeddings.
20 . The apparatus of claim 11 , wherein the graph is a fully-connected graph.