Method for managing annotation job, apparatus and system supporting the same

1. An annotation job management method performed by a computing device, comprising:

obtaining, by a processor of the computing device, information about a medical slide image through a communication interface of the computing device;

inputting, by the processor, the medical slide image to a machine learning model;

determining, by the processor, a dataset type and a panel of the medical slide image based on an output value outputted as a result of the machine learning model;

generating, by the processor, an annotation job based on the medical slide image, the determined dataset type, an annotation task, and a patch that is a partial area of the medical slide image; and

assigning, by the processor, the annotation job to an annotator account through the communication interface,

wherein the panel is determined among a plurality of panels including a cell panel, a tissue panel, and a structure panel,

wherein the dataset type indicates a use of the medical slide image and the use of the medical slide image is determined among a plurality of uses including a training use of the machine learning model and a validation use of the machine learning model, and

wherein determining, by the processor, the dataset type and the panel comprises determining the dataset type of the medical slide image as the training use in response to the output value satisfying a first condition, and determining the dataset type of the medical slide image as the validation use in response to the output value satisfying a second condition.

2. The method of claim 1 , wherein the annotation task includes a task class that indicates an annotation target defined from a perspective of the panel.

3. The method of claim 1 , wherein the plurality of uses further include an OPT (Observer Performance Test) use of the machine learning model.

4. The method of claim 1 , wherein the output value includes a confidence score, and

wherein the first condition includes a condition that the confidence score is less than a reference value.

5. The method of claim 1 , wherein the machine learning model includes a first model corresponding to the cell panel, a second model corresponding to the tissue panel, and a third model corresponding to the structure panel, and

wherein determining, by the processor, the dataset type and the panel of the medical slide image based on the output value comprises:

inputting, by the processor, the medical slide image or the partial area of the medical slide image to each of the first model, the second model, and the third model to obtain the output value outputted as a result; and

determining, by the processor, the panel of the medical slide image based on the output value.

6. The method of claim 1 , wherein obtaining, by the processor, the information about the medical slide image through the communication interface comprises:

detecting, by a worker agent monitoring a storage, that a file of the medical slide image is added to the storage of a designated location,

storing, by the worker agent, information about the medical slide image into a database; and

obtaining, by the processor, the information about the medical slide image from the database through the communication interface.

7. The method of claim 1 , wherein assigning, by the processor, the annotation job to the annotator account through the communication interface comprises

selecting, by the processor, the annotator account based on an annotation performance history associated with a combination of the dataset type and the panel; and

automatically assigning, by the processor, the annotation job to the selected annotator account.

8. The method of claim 1 , wherein the annotation task further includes a task class that indicates an annotation target defined from a perspective of the panel, and

wherein assigning, by the processor, the annotation job to the annotator account through the communication interface comprises

selecting, by the processor, the annotator account based on an annotation performance history associated with a combination of the panel and the task class; and

automatically assigning, by the processor, the annotation job to the selected annotator account.

9. The method of claim 1 , wherein assigning, by the processor, the annotation job to the annotator account through the communication interface comprises:

obtaining, by the processor, candidate patches of the medical slide image through the communication interface;

inputting, by the processor, each of the candidate patches to the machine learning model; and

automatically selecting, by the processor, a patch for the annotation job among the candidate patches based on an output value for each class that is outputted as a modeling result of the machine learning model.

10. The method of claim 9 , wherein automatically selecting, by the processor, the patch for the annotation job among the candidate patches based on the output value for each class that is outputted as the modeling result of the machine learning model comprises:

calculating, by the processor, an entropy value using the output value for each class for each of the candidate patches; and

selecting, by the processor, a candidate patch having the entropy value being equal to or greater than a reference value, as the patch for the annotation job.

11. The method of claim 9 , wherein obtaining, by the processor, the candidate patches of the medical slide image through the communication interface comprises

obtaining, by the processor, the candidate patches by uniformly dividing an entire area of the medical slide image through the communication interface.

12. The method of claim 9 , wherein obtaining, by the processor, the candidate patches of the medical slide image through the communication interface comprises obtaining, by the processor, the candidate patches by randomly dividing an entire area of the medical slide image through the communication interface.

13. The method of claim 9 , wherein obtaining, by the processor, the candidate patches of the medical slide image through the communication interface comprises

performing, by the processor, object recognition on an entire area of the medical slide image; and

forming, by the processor, the candidate patches such that a number of objects obtained from the object recognition is greater than a reference value.

14. The method of claim 9 , wherein obtaining, by the processor, the candidate patches of the medical slide image through the communication interface comprises

obtaining, by the processor, the candidate patches that are divided according to a policy determined based on metadata of the medical slide image through the communication interface.

15. The method of claim 1 , wherein assigning, by the processor, the annotation job to the annotator account through the communication interface comprises:

obtaining, by the processor, candidate patches of the medical slide image through the communication interface;

calculating, by the processor, a misprediction probability of the machine learning model for each of the candidate patches; and

selecting, by the processor, as a patch of the annotation job a candidate patch having the calculated misprediction probability being equal to or greater than a reference value.

16. The method of claim 1 , further comprising:

obtaining, by the processor, a first annotation result data from the annotator account assigned the annotation job through the communication interface;

comparing, by the processor, the first annotation result data with a result that is obtained by inputting a patch of the annotation job to the machine learning model; and

reassigning, by the processor, the annotation job to another annotator account through the communication interface when a difference between the two results is greater than a reference value.

17. The method of claim 1 , further comprising: obtaining, by the processor, a first annotation result data from the annotator account assigned the annotation job through the communication interface;

obtaining, by the processor, a second annotation result data from another annotator account through the communication interface; and

disapproving, by the processor, the first annotation result data when a similarity between the first annotation result data and the second annotation result data is less than a reference value.

18. An annotation management apparatus comprising:

a processor;

a memory; and

a communication interface,

wherein the memory stores one or more instructions; and

wherein the processor, by executing the one or more instructions,

obtains information about a medical slide image through the communication interface,

inputting the medical slide image to a machine learning model;

determines a dataset type and a panel of the medical slide image based on an output value outputted as a result of the machine learning model,

generates an annotation job based on the medical slide image, the determined dataset type, an annotation task, and a patch that is a partial area of the medical slide image, and

assigns the annotation job to an annotator account through the communication interface,

wherein the panel is determined among a plurality of panels including a cell panel, a tissue panel, and a structure panel, and

wherein the dataset type indicates a use of the medical slide image and the use of the medical slide image is determined among a plurality of uses including a training use of the machine learning model and a validation use of the machine learning model, and

wherein the processor determines the dataset type of the medical slide image as the training use in response to the output value satisfying a first condition, and determines the dataset type of the medical slide image as the validation use in response to the output value satisfying a second condition.

19. A non-transitory computer-readable medium storing a computer program that, when executed by a processor, causes the processor to:

obtain information about a medical slide image through a communication interface connected to the processor;

input the medical slide image to a machine learning model;

determine a dataset type and a panel of the medical slide image based on an output value outputted as a result of the machine learning model;

generate an annotation job based on the medical slide image, the determined dataset type, an annotation task, and a patch that is a partial area of a medical slide image; and

assign the annotation job to an annotator account through the communication interface,

wherein the panel is determined among a plurality of panels including a cell panel, a tissue panel, and a structure panel,

wherein the dataset type indicates a use of the medical slide image and the use of the medical slide image is determined among a plurality of uses including a training use of the machine learning model and a validation use of a medical learning model, and

wherein determining the dataset type and the panel comprises determining the dataset type of the medical slide image as the training use in response to the output value satisfying a first condition, and determining the dataset type of the medical slide image as the validation use in response to the output value satisfying a second condition.