Method and system for anatomical labels generation
The present disclosure relates to a method and a system for generating anatomical labels of an anatomical structure. The method includes receiving an anatomical structure with an extracted centerline, or a medical image containing the anatomical structure with the extracted centerline; and predicting the anatomical labels of the anatomical structure based on the centerline of the anatomical structure, by utilizing a trained deep learning network. The deep learning network includes a branched network, a Graph Neural Network, a Recurrent Neural Network and a Probability Graph Model, which are connected sequentially in series. The branched network includes at least two branch networks in parallel. The method in the disclosure can automatically generate the anatomical labels of the whole anatomical structure in medical image end to end and provide high prediction accuracy and reliability.
1. A computer-implemented method for generating anatomical labels of an anatomical structure, comprising:
receiving an anatomical structure with an extracted centerline, or a medical image containing the anatomical structure with the extracted centerline; and
predicting, by at least one processor, the anatomical labels of the anatomical structure based on the centerline of the anatomical structure, by utilizing a trained deep learning network, wherein the deep learning network comprises a branched network, a Graph Neural Network, a Recurrent Neural Network and a Probability Graph Model, which are connected sequentially in series, wherein the branched network comprises at least two branch networks in parallel.
2. The computer-implemented method according to claim 1 , wherein predicting the anatomical labels of the anatomical structure based on the centerline of the anatomical structure, by utilizing a trained deep learning network further comprises:
sampling the centerline to form a centerline graph of the anatomical structure, wherein each sampling point is a node of the centerline graph and each line segment on the centerline connecting each pair of adjacent nodes is an edge of the centerline graph; and
predicting the anatomical labels of the anatomical structure based on the centerline graph, by utilizing the trained deep learning network.
3. The computer-implemented method according to claim 2 , wherein the at least two branch networks include a first branch network and a second branch network, and predicting the anatomical labels of the anatomical structure based on the centerline graph, by utilizing the trained deep learning network further comprises:
generating a geometric feature embedding of each node based on coordinate information of each node, by utilizing the first branch network;
generating an image feature embedding of each node based on an image patch or an mask patch corresponding to each node, by utilizing the second branch network;
generating a joint feature embedding of each node based on the geometric feature embedding and the image feature embedding, by utilizing the Graph Neural Network;
generating an anatomical label corresponding to each node in the centerline graph based on the joint feature embedding, by utilizing the Recurrent Neural Network; and
generating the anatomical labels of the anatomical structure based on the anatomical label corresponding to each node in the centerline graph, by utilizing the Probability Graph Model.
4. The computer-implemented method according to claim 3 , wherein predicting the anatomical labels of the anatomical structure based on the centerline graph, by utilizing the trained deep learning network further comprises:
dividing the centerline into a plurality of cells;
after generating a joint feature embedding of each node by utilizing the Graph Neural Network, performing a cell level average pooling for the joint feature embedding of each node based on the divided cells of the centerline to generate cell level features;
generating the cell level anatomical labels of the centerline graph based on the cell level features, by utilizing the Recurrent Neural Network; and
generating the anatomical labels of the anatomical structure based on the cell level anatomical labels, by utilizing the Probability Graph Model.
5. The computer-implemented method according to claim 4 , wherein the anatomical structure is a vessel, and each cell of the centerline corresponds to a vessel branch between two bifurcations of the vessel.
6. The computer-implemented method according to claim 3 , wherein the first branch network is a point cloud neural network.
7. The computer-implemented method according to claim 3 , wherein the second branch. network is a CNN, a RNN, or an MLP.
8. The computer-implemented method according to claim 3 , wherein the Recurrent Neural Network is an LSTM, a GRU, a GLSTM, or a CGRU.
9. The computer-implemented method according to claim 2 , wherein the edges of the centerline graph are directed edges.
10. The computer-implemented method according to claim 1 , wherein the anatomical structure comprises at least one of a vessel, a respiratory tract, or a mammary duct.
11. The computer-implemented method according to claim 1 , further comprising:
receiving training data comprising a sample anatomical structure with an extracted sample centerline or a sample medical image containing the sample anatomical structure with the extracted sample centerline and a ground truth of the anatomical labels of the sample anatomical structure; and
training the deep learning network by performing a joint optimization of model parameters of the at least two branch networks, the Graph Neural Network, the Recurrent Neural Network and the Probability Graph Model using the training data.
12. A system for generating anatomical labels of an anatomical structure, comprising:
an interface configured to receive an anatomical structure with an extracted centerline, or a medical image containing the anatomical structure with the extracted centerline; and
at least one processor configured to predict the anatomical labels of the anatomical structure based on the centerline of the anatomical structure, by utilizing a trained deep learning network, the deep learning network comprises a branched network, a Graph Neural Network, a Recurrent Neural Network and a Probability Graph Model which are connected sequentially in series, wherein the branched network comprises at least two branch networks in parallel.
13. The system according to claim 12 , wherein to predict the anatomical labels of the anatomical structure based on the centerline of the anatomical structure, by utilizing a trained deep learning network, the at least one processor is further configured to:
sample the centerline to form a centerline graph of the anatomical structure, wherein each sample point is a node of the centerline graph and each line segment on the centerline connecting each pair of adjacent nodes is an edge of the centerline graph; and
predict the anatomical labels of the anatomical structure based on the centerline graph, by utilizing the trained deep learning network.
14. The system according to claim 13 , wherein the at least two branch networks include a first branch network and a second branch network, and to predict the anatomical labels of the anatomical structure based on the centerline graph, by utilizing the trained deep learning network, the at least one processor is further configured to:
generate a geometric feature embedding of each node based on coordinate information of each node, by utilizing the first branch network;
generate an image feature embedding of each node based on an image patch or an mask patch corresponding to each node, by utilizing the second branch network;
generate a joint feature embedding of each node based on the geometric feature embedding and the image feature embedding, by utilizing the Graph Neural Network;
generate an anatomical label corresponding to each node in the centerline graph based on the joint feature embedding, by utilizing the Recurrent Neural Network; and
generate the anatomical labels of the anatomical structure based on the anatomical label corresponding to each node in the centerline graph, by utilizing the Probability Graph Model.
15. The system according to claim 14 , wherein to predict the anatomical labels of the anatomical structure based on the centerline graph, by utilizing the trained deep learning network, the at least one processor is further configured to:
divide the centerline into a plurality of cells;
generate a joint feature embedding of each node by utilizing the Graph Neural Network, performing a cell level average pooling for the joint feature embedding of each node based on the divided cells of the centerline to generate cell level features;
generate the cell level anatomical labels of the centerline graph based on the cell level features, by utilizing the Recurrent Neural Network; and
generate the anatomical labels of the anatomical structure based on the cell level anatomical labels, by utilizing the Probability Graph Model.
16. The system according to claim 15 , wherein the anatomical structure is a vessel, and each cell of the centerline is a vessel branch between two bifurcations of the vessel.
17. The system according to claim 14 , wherein the first branch network is a point cloud neural network.
18. The system according to claim 14 , wherein the second branch network is a CNN, a RNN, or an MLP.
19. The system according to claim 14 , wherein the Recurrent Neural Network is an LSTM, a GRU, a GLSTM, or a CGRU.
20. A non-transitory computer-readable storage medium storing computer-executable instructions thereon, wherein the computer-executable instructions, when executed by a processor, cause the processor to perform a method for generating anatomical labels of an anatomical structure, the method comprising:
receiving an anatomical structure with an extracted centerline, or a medical image containing the anatomical structure with the extracted centerline; and
predicting, by at least one processor, the anatomical labels of the anatomical structure based on the centerline of the anatomical structure, by utilizing a trained deep learning network, wherein the deep learning network comprises a branched network, a Graph Neural Network, a Recurrent Neural Network and a Probability Graph Model, which are connected sequentially in series, wherein the branched network comprises at least two branch networks in parallel.