Method for inspecting labeling on bounding box by using deep learning model and apparatus using same
According to the present invention, proposed is a method for inspecting a labeling operation, the method comprising, when a deep learning model for inspecting a labeling operation for a bounding box corresponding to an object included in an image is present and a computing apparatus uses the deep learning model, the steps of: performing, by the computing apparatus, first training on the deep learning model on the basis of a training image; obtaining, by the computing apparatus, an operation image and a bounding box labeling value therefor; calculating, by the computing apparatus, a score for inspection by performing a calculation while passing the operation image and the bounding box labeling value through the deep learning model; and determining, by the computing apparatus, whether the bounding box labeling value for the operation image is accurate on the basis of the score for inspection and performing any one of a pass process, a fail process, and a re-inspection process.
1. A method for inspecting a labeling operation, in which a deep learning model for inspecting the labeling operation for a bounding box corresponding to an object included in an image is present, and a computing apparatus uses the deep learning model, the method comprising:
(a) performing, by the computing apparatus, first training on the deep learning model based on a training image;
(b) obtaining, by the computing apparatus, an operation image and a bounding box labeling value for the operation image;
(c) calculating, by the computing apparatus, an inspection score by performing calculation while passing the operation image and the bounding box labeling value through the deep learning model; and
(d) determining, by the computing apparatus, whether the bounding box labeling value for the operation image is accurate based on the inspection score, and performing any one of pass processing, fail processing, and re-inspection processing,
wherein, upon a first pair of threshold values provided where a first threshold value of the pair is greater than a second threshold value of the pair, the computing apparatus is configured to:
perform the pass processing on the operation image and the bounding box labeling value when the inspection score is greater than or equal to the first threshold value;
perform the re-inspection processing on the operation image and the bounding box labeling value when the inspection score is less than the first threshold and greater than or equal to the second threshold value; and
perform the fail processing on the operation image and the bounding box labeling value when the inspection score is less than the second threshold value; and
wherein, the computing apparatus determines who is qualified to perform the re-inspection processing according to various levels of the labeling operation difficulty by (1) arranging a second pair of threshold values within the first and second threshold values and (2) finding a location of the inspection score within a particular range of the threshold values of the first and second pairs, in which third and fourth threshold values of the second pair are less than the first threshold value and greater than the second threshold value, respectively, and the third threshold value is greater than the fourth threshold value.
2. The method of claim 1 , wherein the computing apparatus is configured to transmit the operation image and the bounding box labeling value, which are subject to the re-inspection processing, to an operator terminal so as to support re-inspection to be performed in the operator terminal, and perform second training on the deep learning model based on a result of the re-inspection.
3. The method of claim 2 , wherein the computing apparatus is configured to adjust the first threshold value and the second threshold value based on the result of the re-inspection.
4. The method of claim 1 , wherein, a plurality of operators are classified based on corresponding operation performance values,
when at least a p th threshold value and a q th threshold value that are less than the first threshold value and greater than the second threshold value are present, and the p th threshold value is greater than the q th threshold value,
the computing apparatus is configured to:
i) support re-inspection to be performed in a terminal of an operator having a low grade when the inspection score is less than the first threshold value and greater than or equal to the p th threshold value, or when the inspection score is less than the q th threshold value and greater than or equal to the second threshold value; and
ii) support the re-inspection to be performed in a terminal of an operator having a high grade when the inspection score is less than the p th threshold value and greater than or equal to the q th threshold value.
5. The method of claim 1 , wherein the inspection score is a probability value corresponding to a, and satisfies 0<=a<=1.
6. The method of claim 1 , wherein, in the step (a),
while at least one parameter is present to perform the calculation of the deep learning model,
the computing apparatus is configured to:
i) adjust the at least one parameter so that a result of performing the calculation while passing the training image and a correct answer bounding box labeling value through the deep learning model is 1; and
ii) adjust the at least one parameter so that a result of performing the calculation while passing the training image and an incorrect answer bounding box labeling value through the deep learning model is 0.
7. A computing apparatus, which is an apparatus for inspecting a labeling operation, in which a deep learning model for inspecting the labeling operation for a bounding box corresponding to an object included in an image is present, and the computing apparatus uses the deep learning model, the computing apparatus comprising:
a communication unit for transmitting and receiving information to and from an operation terminal; and
a processor for performing first training on the deep learning model based on a training image, obtaining an operation image and a bounding box labeling value for the operation image, calculating an inspection score by performing calculation while passing the operation image and the bounding box labeling value through the deep learning model, determining whether the bounding box labeling value for the operation image is accurate based on the inspection score, and performing any one of pass processing, fail processing, and re-inspection processing,
wherein, upon a first pair of threshold values provided where a first threshold value of the pair is greater than a second threshold value of the pair, the processor is configured to:
perform the pass processing on the operation image and the bounding box labeling value when the inspection score is greater than or equal to the first threshold value;
perform the re-inspection processing on the operation image and the bounding box labeling value when the inspection score is less than the first threshold and greater than or equal to the second threshold value; and
perform the fail processing on the operation image and the bounding box labeling value when the inspection score is less than the second threshold value; and
wherein, the computing apparatus determines who is qualified to perform the re-inspection processing according to various levels of the labeling operation difficulty by (1) arranging a second pair of threshold values within the first and second threshold values and (2) finding a location of the inspection score within a particular range of the threshold values of the first and second pairs, in which third and fourth threshold values of the second pair are less than the first threshold value and greater than the second threshold value, respectively, and the third threshold value is greater than the fourth threshold value.