MULTI-FRAME ANALYSIS FOR CLASSIFYING TARGET FEATURES IN MEDICAL VIDEOS
Methods, systems, and devices for classifying a target feature in a medical video are presented herein. Some methods may include the steps of: receiving a plurality of frames of the medical video, where the plurality of frames include the target feature; generating, by a first pretrained machine learning model, an embedding vector for each frame of the plurality of frames, each embedding vector having a predetermined number of values; and generating, by a second pretrained machine learning model, a classification of the target feature using the plurality of embedding vectors, where the second pretrained machine learning model analyzes the plurality of embedding vectors jointly.
1 . A method of classifying a target feature in a medical video by one or more computer systems, wherein the one or more computer systems comprises a first pretrained machine learning model and a second pretrained learning model, the method comprising:
receiving a plurality of frames of the medical video, wherein the plurality of frames comprises the target feature;
generating, by the first pretrained machine learning model, an embedding vector for each frame of the plurality of frames, each embedding vector having a predetermined number of values; and,
generating, by the second pretrained machine learning model, a classification of the target feature using the plurality of embedding vectors, wherein the second pretrained machine learning model analyzes the plurality of embedding vectors jointly.
2 . The method of claim 1 , wherein the first pretrained learning model comprises a convolutional neural network, and wherein the second pretrained machine learning model comprises a transformer.
3 . The method of claim 1 , wherein the classification comprises a score, wherein the score is in a range of 0 to 1.
4 . The method of claim 1 , wherein the classification comprises one of: positive, negative, or uncertain.
5 . The method of claim 1 , wherein the classification comprises a textual representation.
6 . The method of claim 1 , wherein the first pretrained machine learning model and the second pretrained machine learning model are jointly trained.
7 . The method of claim 1 , wherein the first pretrained machine learning model and the second pretrained machine learning model are trained separately.
8 . The method of claim 1 , wherein the medical video is collected during a colonoscopy procedure using an endoscope and wherein the target feature is a polyp.
9 . The method of claim 8 , wherein the classification comprises one of: adenomatous and non-adenomatous.
10 . The method of claim 1 , wherein the second pretrained machine learning model analyzes the plurality of embedding vectors without classifying each embedding vector individually.
11 . A system for classifying a target feature in a medical video comprising:
an input interface configured to receive a medical video;
a memory configured to store a plurality of processor-executable instructions, the memory including:
an embedder based on a first pretrained machine learning model; and,
a classifier based on a second pretrained machine learning model; and,
a processor configured to execute the plurality of processor-executable instruction to perform operations including:
receiving a plurality of frames of the medical video, wherein the plurality of frames comprises the target feature;
generating, with the embedder, an embedding vector for each frame of the plurality of frames, each embedding vector having a predetermined number of values; and,
generating, with the classifier, a classification of the target feature using the plurality of embedding vectors, wherein the classifier analyzes the plurality of embedding vectors jointly.
12 . The system of claim 11 , wherein the first pretrained machine learning model comprises a convolutional neural network and the second pretrained machine learning model comprises a transformer.
13 . The system of claim 11 , wherein the classification comprises a score, wherein the score is in a range of 0 to 1.
14 . The system of claim 11 , wherein the classification comprises one of: positive, negative, or uncertain.
15 . The system of claim 14 , wherein the classification comprises a textual representation.
16 . A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for classifying a target feature in a medical video, the instructions being executed by a processor to perform operations comprising:
receiving a plurality of frames of the medical video, wherein the plurality of frames comprises the target feature;
generating, by a first pretrained machine learning model, an embedding vector for each frame of the plurality of frames, each embedding vector having a predetermined number of values; and,
generating, by a second pretrained machine learning model, a classification of the target feature using the plurality of embedding vectors, wherein the second pretrained machine learning model analyzes the plurality of embedding vectors jointly.
17 . The non-transitory processor-readable storage medium of claim 16 , wherein the first pretrained machine learning model comprises a convolutional neural network and the second pretrained machine learning model comprises a transformer.
18 . The non-transitory processor-readable storage medium of claim 16 , wherein the classification comprises a score, wherein the score is in a range of 0 to 1.
19 . The non-transitory processor-readable storage medium of claim 16 , wherein the classification comprises one of: positive, negative, or uncertain.
20 . The non-transitory processor-readable storage medium of claim 19 , wherein the classification comprises a textual representation.