Systems and methods for polygon object annotation and a method of training and object annotation system
The present invention relates generally to object annotation, specifically to polygonal annotations of objects. Described are methods of annotating an object including steps of receiving an image depicting an object, generating a set of image features using a CNN encoder implemented on one or more computers, and producing a polygon object annotation via a recurrent decoder or a Graph Neural Network. The recurrent decoder may include a recurrent neural network, a graph neural network or a gated graph neural network. A system for annotating an object and a method of training an object annotation system are also described.
1. A method of annotating an object, comprising:
receiving an image depicting an object, the image comprising an n-dimensional array-like data structure;
generating a set of image features using a Convolutional Neural Network (CNN) encoder implemented on one or more computers;
initializing a set of N nodes from the set of image features, the set of N nodes forming a closed curve along a circle centered in the image;
predicting a location shift for each node simultaneously using a Graph Neural Network (GNN);
iterating predictions through the GNN for each node, each iteration defining a new location shift for each node based on node locations for each node from the previous iteration; and
producing an object annotation based on a final iteration;
wherein the object is parametrized with one of polygons or splines.
2. The method of claim 1 , further comprising applying one or more simulated human corrections or human corrections to one or more nodes during the iteration step.
3. The method of claim 1 , wherein the GNN is trained via point matching loss.
4. The method of claim 3 , wherein the GNN is further trained and fine-tuned via differentiable accuracy loss.
5. The method of claim 1 , wherein a centripetal Catmull-Rom spline is applied for splines.
6. The method of claim 1 , wherein the image is selected from one of: an RGB image, a thermal image, a depth image, and a hyperspectral image.
7. The method of claim 1 , wherein the final iteration is based on running a pre-selected number of iterations.
8. The method of claim 1 , wherein the GNN is a Graph Convolutional Network.
9. A system for object annotation, comprising:
a Convolutional Neural Network (CNN), encoder implemented by one or more computers for generating a set of image features from a received image, the image comprising an n-dimensional array-like data structure, and for initializing a set of N nodes from the set of image features, the set of N nodes forming a closed curve along a circle centered in the image;
a Graph Neural Network (GNN) implemented by one or more computers for predicting a location shift for each node simultaneously and iterating predictions through the GNN for each node, each iteration defining a new location shift for each node based on node locations for each node from the previous iteration; and
an output selector for producing an output based on a final iteration from the GNN.
10. The system of claim 9 , further comprising a human input interface for receiving one or more human corrections to one or more nodes during the iteration.
11. The system of claim 9 , further comprising a training system implemented by one or more computers for training the GNN.
12. The system of claim 11 , wherein the training system applies a point matching loss training method or a differentiable accuracy loss method.
13. The system of claim 11 , wherein the training system applies simulated human corrections or human corrections.
14. The system of claim 11 , wherein the final iteration is based on running a pre-selected number of iterations.
15. The system of claim 9 , wherein the GNN is a Graph Convolutional Network.