Method for labelling a training set involving a GUI
The present invention relates to a computer-implemented method for labelling a training set, preferably for training a neural network, with respect to a 3D physical object by means of a GUI, the method comprising the steps of: obtaining a training set relating to a plurality of training objects, each of the training objects comprising a 3D surface similar to the 3D surface of said object, the training set comprising at least two images for each training object; generating, for each training object, a respective 3D voxel representation based on the respective at least two images; receiving, via said GUI, manual annotations with respect to a plurality of segment classes from a user of said GUI for labelling each of the training objects; and preferably training, based on said manual annotations ( 91, 92, 93 ), at least one NN, for obtaining said at least one trained NN.
1. A computer-implemented method for labelling a training set, for training a NN, with respect to a 3D physical object by a Graphical User Interface, GUI, the method comprising the steps of:
obtaining the training set relating to a plurality of training objects, each of the training objects comprising a 3D surface similar to the 3D surface of said object, the training set comprising at least two images for each training object;
generating, for each training object, a respective 3D voxel representation based on the respective at least two images;
receiving, via said GUI, manual annotations with respect to a plurality of segment classes from a user of said GUI for labelling each of the training objects; and
training, based on said manual annotations, at least one NN, for obtaining said at least one trained NN;
wherein the GUI comprises
a 3D reconstruction view showing said 3D voxel representation for a respective one of the plurality of training objects; and
at least one image view showing at least one of the respective at least two images associated with the respective training object; said at least one image view being displayed along with said 3D reconstruction view;
and wherein the method comprises the further step of:
for at least one training object, and based on the manual annotation of at least one of
a first image belonging to said at least two images, via said at least one image view, and
said 3D voxel representation, via said 3D reconstruction view, and automatically annotate the other one of said first image and said 3D voxel representation, based on automatically projecting the manual annotation of the at least one of the first image and the 3D voxel representation to the other one of said first image and said 3D voxel representation.
2. The computer-implemented method of claim 1 , wherein the GUI is configured to let the receipt of the manual annotation from the user via at least one of said 3D reconstruction view and said at least one image view cause the other one of said 3D reconstruction view and one of said at least one image view to be updated according to said automatic annotation.
3. The computer implemented method of 1 , wherein said automatic annotation is performed based on manual annotation of either of said first image or said 3D voxel representation to automatically annotate the other one of said first image and said 3D reconstruction view.
4. The computer implemented method of claim 3 , wherein the GUI is configured to let said receipt of the manual annotation from the user via either one of said 3D reconstruction view and one of said at least one image view cause the other one of said 3D reconstruction view and one of said at least one image view to be updated according to said automatic annotation.
5. The computer implemented method of claim 4 , further comprising the step of:
for at least one training object, and based on the manual annotation of at least said first image belonging to said at least two images, automatically annotate at least one second image belonging to said at least two images and different from said at least one first image, based on automatically projecting the manual annotation of the at least one first image to the 3D voxel representation and back to the second image.
6. The computer implemented method of claim 1 , wherein the GUI is configured to let the receipt of the manual annotation cause each of the at least one image view and the 3D reconstruction view to be updated according to said automatic annotation.
7. The computer implemented method of claim 1 , wherein the GUI comprises at least two image views each showing one of the respective at least two images, and wherein each of said at least two image views is displayed along with said 3D reconstruction view.
8. The computer implemented method of claim 1 , wherein for each training object, the respective at least two images originate from a plurality of respective predetermined camera angles and respective predetermined camera positions with respect to said training object, said predetermined camera angles and predetermined camera positions being the same for each training object.
9. The computer implemented method of claim 8 , wherein said predetermined camera positions and predetermined camera angles relate to respective ones of a plurality of cameras with fixed relative positions.
10. The computer implemented method of claim 1 , wherein said plurality of respective segment classes are associated with a respective highlighting type, each highlighting type relating at least to a distinct color, and wherein the GUI is configured to let the receipt of the manual annotation relating to a first segment class via at least one of said 3D reconstruction view and said at least one image view cause the each of said 3D reconstruction view and one of said at least one image view to be updated in accordance with the highlighting type associated with said first segment class.
11. The computer implemented method of claim 1 , wherein said training of the NN is directed at least at discriminating between each of said plurality of segment classes, and wherein the NN comprises a semantic segmentation NN.
12. The computer implemented method of claim 11 , wherein said semantic segmentation NN comprises any or any combination of: 2D U-net, 3D U-net, Dynamic Graph CNN, DGCNN, PointNet++.
13. The computer implemented method of claim 1 , wherein the training of the NN is directed at least at discriminating between each of two or more instances of said 3D physical object or two or more instances of a portion of said 3D physical object, and wherein the NN comprises an instance segmentation NN.
14. The computer implemented method of claim 13 , wherein said instance segmentation NN comprises any or any combination of: Mask R-CNN, DeepMask, TensorMask, 3D-BoNet, ASIS.
15. A labelled training set or a trained NN obtained by the computer-implemented method according to claim 1 .