Compression of catalogue of surgical video
Data captured during a surgical procedure can include multiple video streams, such as from an endoscopic camera, an external camera, etc., along with data from one or more instruments used during the surgical procedure. Transferring the surgical data over communication networks, and storing the surgical data, is therefore resource-intensive, and a technical challenge. Technical solutions are described to optimize the surgical data by intelligent compression methods that can adaptively reduce the size of certain procedural workflows. In some aspects, the compression reduces the storage/transmission resources required for a video catalogue by reducing redundancy from across the several videos stored in the video catalogue. The compression also enables faster querying and searching through a catalogue or collection of archived data.
1 . A computer-implemented method comprising:
segmenting, by a processor, a first video of a surgical procedure into a sequence of video portions;
generating, by the processor, a first sequence of latent representations corresponding to the sequence of video portions in the first video using an encoder machine learning model;
computing, by the processor, a similarity score of the first video and a second video from a video catalogue that comprises a plurality of videos, including the second video, the computing based on the first sequence of latent representations and a second sequence of latent representations of the second video;
determining, by the processor, a compression rate for the first video based on the similarity score;
generating, by the processor, a compressed first video by compressing the first video using the compression rate that is determined; and
storing, by the processor, the compressed first video in the video catalogue.
2 . The computer-implemented method of claim 1 , wherein the first video is stored in an archive, and the video catalogue is updated to store a link between the compressed first video and the first video in the archive.
3 . The computer-implemented method of claim 1 , wherein the first video is segmented, using one or more machine learning models, into the sequence of video portions, wherein each video portion represents a maneuver in the surgical procedure.
4 . The computer-implemented method of claim 1 , wherein the first video is captured using a camera that is one from a group comprising an endoscopic camera, a portable camera, and a stationary camera.
5 . The computer-implemented method of claim 1 , wherein the second video is stored using a first compression rate, the first compression rate providing high-fidelity, and based on the similarity score being within a predetermined range, using a second compression rate for the first video, the second compression rate providing a lower fidelity than the second video.
6 . The computer-implemented method of claim 5 , wherein the second video is of the same surgical procedure as the first video.
7 . The computer-implemented method of claim 5 , wherein the second video is of a different surgical procedure than the first video.
8 . The computer-implemented method of claim 1 , wherein the first video and the second video capture respective surgical procedures, and the second video is selected to be compared with the first video based on one or more attributes selected from a group of attributes comprising type of the surgical procedure, institution of the surgical procedure, staff performing the surgical procedure, equipment used for the surgical procedure, patient of the surgical procedure, and camera used to capture the first video.
9 . The computer-implemented method of claim 2 , further comprising:
in response to receiving, from a user, a request to playback the first video from the video catalogue, notifying the user of the compressed first video, the first video from the archive, and the second video from the video catalogue.
10 . The computer-implemented method of claim 9 , wherein the user can select, for playback, one from the compressed first video, the first video from the archive, and the second video from the video catalogue.
11 . A system comprising:
a machine learning system comprising one or more machine learning models that are trained to encode a portion of video into a latent representation; and
a data collection system configured to generate a compressed copy of a video catalogue that comprises a plurality of videos, each video in the video catalogue comprising a plurality of video portions, wherein generating the compressed copy of the video catalogue comprises:
generating a first sequence of latent representations corresponding to a first sequence of video portions in a first video using the machine learning system;
computing a plurality of similarity scores of the first video with the plurality of videos from the video catalogue, a similarity score between the first video and a second video from the plurality of videos is computed based on the first sequence of latent representations and a second sequence of latent representations of the second video;
based on a determination that the first video is similar to the second video based on the similarity score, generating a compressed first video using a compression protocol that is based on the similarity score; and
storing the compressed first video in the video catalogue.
12 . The system of claim 11 , wherein the plurality of similarity scores is stored in metadata of the video catalogue.
13 . The system of claim 11 , wherein the plurality of similarity scores is stored in metadata of the compressed first video.
14 . The system of claim 11 , wherein the first video is archived, and the compressed first video is linked to the first video that is archived.
15 . The system of claim 11 , wherein, in response to the similarity score being within a first range that indicates that the first video is similar to the second video, adjusting a compression rate to a higher value, and in response to the similarity score being within a second range that indicates that the first video is not similar to the second video, adjusting the compression rate to a lower value.
16 . The system of claim 15 , wherein adjusting the compression rate comprises adjusting one or more attributes from a group comprising image size, frame rate, amount of movement between frames, bit rate, and codec.
17 . A computer program product comprising a memory device having computer-executable instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform a method to catalogue surgical data in a compressed manner, the method comprising:
generating, using a machine learning system, a latent representation space corresponding to a data collection system that stores surgical data for a plurality of surgical procedures, the latent representation space comprising a plurality of latent representations, wherein a latent representation is a vector representation of a portion of surgical data, and wherein each surgical data in the data collection system comprises a plurality of portions; and
in response to receiving a first surgical data to be catalogued in the data collection system:
segmenting the first surgical data into a sequence of portions;
generating a first sequence of latent representations corresponding to the sequence of portions in the first surgical data using the machine learning system;
determining, from the latent representation space, a second surgical data that is similar to the first surgical data by comparing the first sequence of latent representations of the first surgical data and a second sequence of latent representations of the second surgical data;
computing a similarity score of the first surgical data and the second surgical data;
determining a first compression rate by adjusting a second compression rate based on the similarity score, the second compression rate is used to store the second surgical data in the data collection system;
generating a compressed first surgical data using the first compression rate; and
storing the compressed first surgical data in the data collection system.
18 . The computer program product of claim 17 , wherein, the first compression rate is higher than the second compression rate.
19 . The computer program product of claim 17 , wherein storing the compressed first surgical data in the data collection system comprises storing a link between the compressed first surgical data and the second surgical data.
20 . The computer program product of claim 17 , wherein:
in response to receiving a third surgical data to be catalogued in the data collection system:
segmenting the third surgical data into a third sequence of portions;
generating a third sequence of latent representations corresponding to the third sequence of portions in the third surgical data using the machine learning system;
determining, based on the latent representation space, that the third surgical data is not similar to any of the surgical data in the data collection system;
storing the third surgical data in the data collection system using the second compression rate; and
storing the third sequence of latent representations in the latent representation space.