Deep learning method, apparatus, device, and medium for joint optimization based on vascular centerline guidance
This disclosure provides a deep learning method, apparatus, device, and medium for task joint optimization based on vascular centerline guidance, including: in a process of extracting a global image feature by an encoder, extracting a local image feature after performing data augmentation on a local image with morphological specificity based on the vascular centerline guidance, and fusing the local image feature with the global image feature to form a fused image feature; and forming a bi-directional feature correction process between decoders of different decoding branches. The vascular centerline guidance is used to provide gated feature optimization and computational efficiency adjustment for an encoding process. Specific morphological data of the vascular centerline guidance is used to form a control mechanism for modulating efficiency of local feature extraction. A scarcity problem of hemodynamic annotations has been alleviated and generalization ability of a model has been enhanced.
1 . A deep learning method for joint optimization based on vascular centerline guidance, comprising:
in a process of extracting a global image feature by an encoder, extracting a local image feature after performing data augmentation on a local image with morphological specificity based on the vascular centerline guidance, and fusing the local image feature with the global image feature to form a fused image feature; and
forming a bi-directional feature correction process between decoders of different decoding branches.
2 . The deep learning method according to claim 1 , wherein the forming the bi-directional feature correction process between the decoders of different decoding branches comprises:
configuring a decoder based on a task type to decode the fused image feature, and forming at least one decoding branch for a vascular morphological task and at least one decoding branch for a hemodynamic functional task; and
in any pair of decoding branches in two task types, forming the bi-directional feature correction process using an output feature of one decoder to correct an output feature of the other decoder between the decoders of the two task types.
3 . The deep learning method according to claim 1 , wherein the forming the fused image feature comprises:
performing global sampling on an input image containing a blood vessel to form a global ROI based on the vascular centerline guidance;
extracting the global image feature from the global ROI by the encoder;
determining position attention information based on a vascular specificity judgment rule in the global ROI, and performing local sampling to form a local ROI dataset based on the position attention information in the global ROI;
after performing up-sampling and the data augmentation on the local ROI, extracting the local image feature using an additional encoder, and performing positional encoding on the local image feature using a vascular centerline; and
fusing the local image feature with the global image feature using an attention mechanism to form the fused image feature.
4 . The deep learning method according to claim 3 , wherein the vascular specificity judgment rule comprises judging vascular stenosis rate, vascular bifurcation, and vascular curvature.
5 . The deep learning method according to claim 3 , wherein replace the determining the position attention information based on the vascular specificity judgment rule in the global ROI, and the performing the local sampling to form the local ROI dataset based on the position attention information in the global ROI by:
adjusting the local ROI to perform the local sampling to form the local ROI dataset in the global ROI, and determining the position attention information of the local ROI containing specific vascular morphology based on the vascular specificity judgment rule.
6 . The deep learning method according to claim 3 , wherein the up-sampling is provided based on a resolution level of fusion of the local image feature and the global image feature.
7 . The deep learning method according to claim 3 , wherein the additional encoder adopts cascaded 3D convolution to extract the local feature and fuse it with positional encoding of a vascular centerline point to obtain the local image feature.
8 . The deep learning method according to claim 3 , wherein a size of the local ROI is dynamically adjusted based on the vascular specificity judgment rule.
9 . The deep learning method according to claim 3 , wherein the vascular centerline comprises stenosis information, obtained by one of the following methods:
in the case that the centerline is known, obtain a cross-sectional diameter adopting cross-sectional 2D-Unet segmentation, and calculate stenosis degree along the centerline, and in the case that the centerline is unknown, segment an entire image and extract coordinates of the centerline adopting lightweight 3D-Unet, and calculate vascular stenosis along the centerline.
10 . The deep learning method according to claim 2 , wherein the configuring the decoder based on the task type to decode the fused image feature, and the forming the at least one decoding branch for the vascular morphological task and the at least one decoding branch for the hemodynamic functional task comprise:
providing a segmentation decoding branch reflecting the vascular morphological task, and providing a segmentation decoder corresponding to a same level encoder in the segmentation decoding branch, wherein the vascular morphological task comprises at least one of contour, inner diameter, inner wall mask, or topological shape; and
providing a prediction decoding branch reflecting the hemodynamic functional task, and providing a prediction decoder corresponding to a same level encoder in the prediction decoding branch, wherein the hemodynamic functional task comprises at least one of blood pressure parameters, blood flow parameters, and CTFFR.
11 . The deep learning method according to claim 2 , wherein input features of the bi-directional feature correction process comprise a feature output of a same level decoder as a corrected feature, and further comprise a feature output of a same level decoder as a control feature, a feature output of a preceding level decoder, or a down-sampling feature output of a final output result of a task, and then an interactive judgment is formed using the control feature to control the bi-directional feature correction process.
12 . The deep learning method according to claim 2 , wherein the bi-directional feature correction process comprises:
establishing a forward correction process between two decoders corresponding to the task type of the decoding branch to correct a functional feature based on a morphological feature; and
meanwhile, establishing a reverse correction process between the two decoders to correct the morphological feature based on the functional feature.
13 . The deep learning method according to claim 12 , wherein the forward correction process comprises:
generating a spatial attention weight based on a segmentation confidence map of the morphological feature, and dynamically modulating a functional feature map to obtain a corrected functional feature.
14 . The deep learning method according to claim 12 , wherein the reverse correction process comprises:
correcting the morphological feature based on changes in the functional feature (such as in a gradient direction) to obtain a corrected morphological feature.
15 . The deep learning method according to claim 2 , wherein the forming the bi-directional feature correction process comprises:
establishing a forward correction process between the two decoders corresponding to the task type of the decoding branch based on the morphological feature and a vascular centerline distance heatmap to correct the functional feature; and
meanwhile, establishing a reverse correction process between the two decoders to correct the morphological feature based on the functional feature and a hemodynamic direction of the vascular centerline guidance.
16 . The deep learning method according to claim 2 , wherein the forming the bi-directional feature correction process comprises:
providing a measurement threshold for average confidence of the morphological feature during a gated judgment process, and determining whether to activate the bi-directional feature correction process based on a measurement result.
17 . The deep learning method according to claim 16 , wherein the providing the measurement threshold for the average confidence of the morphological feature during the gated judgment process, and the determining whether to activate the bi-directional feature correction process based on the measurement result comprise:
obtaining a morphological feature to be corrected, a functional feature to be corrected, a control segmentation feature, and a control function feature, and forming a confidence map and the average confidence based on the control function feature;
in the case that the average confidence is greater than the measurement threshold, directly outputting the morphological feature to be corrected and the functional feature to be corrected from the segmentation decoder and the prediction decoder, respectively; and
in the case that the average confidence is less than the measurement threshold, activating the bi-directional feature correction process.
18 . The deep learning method according to claim 16 , wherein the forming the bi-directional feature correction process comprises:
outputting a confidence map (Conf Map) of a determined channel and the vascular centerline distance heatmap or performing 3×3×3 convolution on the control function feature (F_control) based on confidence map (Conf Map) and the vascular centerline distance heatmap to form functional correction data;
outputting a functional gradient (FFR Grad) of a same channel and overlaying with the hemodynamic direction of the vascular centerline guidance or performing the 3×3×3 convolution on a control separation feature (S_control) based on the functional gradient (FFR Grad) and the hemodynamic direction of the vascular centerline guidance to form morphological correction data;
correcting the morphological feature to be corrected (S_feat) based on the morphological correction data to form a corrected morphological feature (S_feat_cor); and
correcting the functional feature to be corrected (F_feat) based on the functional correction data to form a corrected functional feature (F_feat_cor).
19 . The deep learning method according to claim 2 , further comprising:
using vascular morphological parameters and hemodynamic functional parameters obtained from the decoding branches of the two task types as independent variables of physical constraint conditions, determining a physical constraint loss function as a component of a total loss function based on a residual between a predicted value of a dependent variable and a true value of the dependent variable under the physical constraint conditions, and optimizing the decoder and the bi-directional feature correction process through gradient backpropagation of the total loss function.
20 . The deep learning method according to claim 19 , wherein the determining the physical constraint loss function as the component of the total loss function, and the optimizing the decoder and the bi-directional feature correction process through the gradient backpropagation of the total loss function comprise:
determining a segmentation loss function based on a residual between a prediction result of a segmentation task and a segmentation true value;
determining a prediction loss function based on a residual between a prediction result of a prediction task and a predicted true value;
selecting a segmentation task and a CTFFR prediction task, using a segmentation task result to obtain vascular radius parameters and obtain pressure differential parameters from a CTFFR prediction task result, using the vascular radius parameters and the pressure differential parameters to obtain a predicted blood flow value through a Poiseuille flow equation, using the vascular radius parameters and the pressure differential parameters to form a vascular blood flow true value through a fluid continuity equation, and determining the physical constraint loss function based on a residual between the predicted blood flow value and the vascular blood flow true value; and
forming a total loss based on segmentation loss, prediction loss, and physical constraint loss, and optimizing network parameters of the decoder and the bi-directional feature correction process through the gradient backpropagation using the total loss function.
21 . A deep learning apparatus for joint optimization based on vascular centerline guidance, comprising:
an encoder configured to, in a process of extracting a global image feature by an encoder, extract a local image feature after performing data augmentation on a local image with morphological specificity based on the vascular centerline guidance, and fuse the local image feature with the global image feature to form a fused image feature; and
a decoder correction module configured to, form a bi-directional feature correction process between decoders of different decoding branches.
22 . The deep learning apparatus according to claim 21 , wherein the decoder correction module comprises:
a decoder configured to, configure a decoder based on a task type to decode the fused image feature, and form at least one decoding branch for a vascular morphological task and at least one decoding branch for a hemodynamic functional task; and
a bi-directional correction decoder configured to, in any pair of decoding branches in two task types, form the bi-directional feature correction process using an output feature of one decoder to correct an output feature of the other decoder between the decoders of the two task types.
23 . The deep learning apparatus according to claim 22 , further comprising:
a physical constraint formation module configured to, use vascular morphological parameters and hemodynamic functional parameters obtained from the decoding branches of the two task types as independent variables of physical constraint conditions, determine a physical constraint loss function as a component of a total loss function based on a residual between a predicted value of a dependent variable and a true value of the dependent variable under the physical constraint conditions, and optimize the decoder and the bi-directional feature correction process through gradient backpropagation of the total loss function.
24 . An electronic device, comprising:
a processor, a memory, and an interface for communication with a gateway, wherein
the memory is configured to store a program and data, and the processor calls the program stored in the memory to perform the method according to claim 1 .
25 . A non-transitory computer-readable storage medium, wherein the computer-readable storage medium comprises a program, which, when executed by a processor, is configured to perform the method according to claim 1 .