Method for managing annotation job, apparatus and system supporting the same

1. An annotation job management method performed by at least one computing device, comprising:

obtaining a medical slide image that is an annotation target;

determining at least one of a dataset type and a panel type of the obtained medical slide image;

selecting at least one annotation job target patch from among a plurality of candidate patches which are included in the medical slide image, based on at least one of the determined dataset type and panel type;

assigning the at least one selected annotation job target patch to at least one annotator account;

obtaining an annotation result for the at least one annotation job target patch from the assigned annotator account; and

training or validating a machine learning model using a dataset generated based on the annotation result,

wherein the panel type indicates at least one of a cell panel, a tissue panel, and a structure panel.

2. The annotation job management method of claim 1 , wherein selecting the at least one annotation job target patch comprises:

selecting the plurality of candidate patches sampled from the medical slide image;

calculates at least one of a confidence score and an entropy value of each of the plurality of candidate patches; and

selecting the at least one annotation job target patch based on the at least one of the confidence score and the entropy value calculated for each of the plurality of candidate patches.

3. The annotation job management method of claim 2 , wherein selecting the plurality of candidate patches comprises:

dividing at least a part of the medical slide image based on information associated with the medical slide image; and

selecting the plurality of candidate patches from the divided at least part.

4. The annotation job management method of claim 1 , wherein selecting the at least one annotation job target patch comprises:

selecting the plurality of candidate patches sampled from the medical slide image;

calculating a misprediction probability of the machine learning model for each of the plurality of candidate patches; and

selecting the at least one annotation job target patch from among the plurality of candidate patches based on the calculated misprediction probability.

5. The annotation job management method of claim 1 , wherein assigning the at least one annotation job target patch to the at least one annotator account comprises assigning the at least one annotation job target patch to the at least one annotator account based on at least one of the determined dataset type and panel and an annotation performance history of an annotator.

6. The annotation job management method of claim 1 , further comprising:

comparing the obtained annotation result with a result of the machine learning model for the at least one annotation job target patch; and

determining whether to reassign the at least one annotation job target patch based on a compared result.

7. The annotation job management method of claim 1 , wherein the at least one annotator account includes a plurality of annotator accounts,

wherein obtaining the annotation result includes obtaining annotation results for the at least one annotation job target patch from the plurality of annotator accounts assigned for the at least one annotation job target patch, and

wherein the annotation job management method further comprises:

comparing the annotation results of the plurality of annotator accounts; and

determining whether to reassign the at least one annotation job target patch based on a compared result.

8. The annotation job management method of claim 1 ,

wherein the dataset type indicates a use of the medical slide image, and the use of the medical slide image includes at least one of a training use of the medical learning model and a validation use of the machine learning model.

9. The annotation job management method of claim 1 , wherein determining the at least one comprises determining at least one of the dataset type and panel type of the medical slide image based on an output value outputted by inputting the medical slide image to the machine learning model.

10. An annotation job management apparatus comprising:

a memory including one or more instructions; and

a processor that, by executing the one or more instructions:

obtains a medical slide image that is an annotation target;

determines at least one of a dataset type and a panel type of the obtained medical slide image;

selects at least one annotation job target patch from among a plurality of candidate patches which are included in the medical slide image, based on at least one of the determined dataset type and panel type;

assigns the at least one selected annotation job target patch to at least one annotator account;

obtains an annotation result for the at least one annotation job target patch from the assigned annotator account; and

trains or validates a machine learning model using a training dataset generated based on the annotation result,

wherein the panel type indicates at least one of a cell panel, a tissue panel, and a structure panel.

11. The annotation job management apparatus of claim 10 , wherein the processor:

selects the plurality of candidate patches sampled from the medical slide image;

calculates at least one of a confidence score and an entropy value of each of the plurality of candidate patches; and

selects the at least one annotation job target patch based on the at least one of the confidence score and the entropy value calculated for each of the plurality of candidate patches.

12. The annotation job management apparatus of claim 11 , wherein the processor:

divides at least a part of the medical slide image based on information associated with the medical slide image; and

selects the plurality of candidate patches from the divided at least part.

13. The annotation job management apparatus of claim 10 , wherein the processor:

selects the plurality of candidate patches sampled from the medical slide image;

calculates a misprediction probability of the machine learning model for each of the plurality of candidate patches; and

selects the at least one annotation job target patch from among the plurality of candidate patches based on the calculated misprediction probability.

14. The annotation job management apparatus of claim 10 , wherein the processor assigns the at least one annotation job target patch to the at least one annotator account based on at least one of the determined dataset type and panel and an annotation performance history of an annotator.

15. The annotation job management apparatus of claim 10 , wherein the processor:

compares the obtained annotation result with a result of the machine learning model for the at least one annotation job target patch; and

determines whether to reassign the at least one annotation job target patch based on a compared result.

16. The annotation job management apparatus of claim 10 , wherein the at least one annotator account includes a plurality of annotator accounts, and

wherein the processor:

obtains annotation results for the at least one annotation job target patch from the plurality of annotator accounts assigned for the at least one annotation job target patch;

compares the annotation results of the plurality of annotator accounts; and

determines whether to reassign the at least one annotation job target patch based on a compared result.

17. The annotation job management apparatus of claim 10 ,

wherein the dataset type indicates a use of the medical slide image, and the use of the medical slide image includes at least one of a training use of the medical learning model and a validation use of the machine learning model.

18. The annotation job management apparatus of claim 10 , wherein the processor determines at least one of the dataset type and panel type of the medical slide image based on an output value outputted by inputting the medical slide image to the machine learning model.