Determining interventional device shape
A computer-implemented method of providing a neural network for predicting a three-dimensional shape of an interventional device disposed within a vascular region, includes: training (S 140 ) a neural network ( 140 ) to predict, from received X-ray image data ( 120 ) and received volumetric image data ( 110 ), a three-dimensional shape of the interventional device constrained by the vascular region ( 150 ). The training includes constraining the adjusting of parameters of the neural network such that the three-dimensional shape of the interventional device predicted by the neural network ( 150 ) fits within the three-dimensional shape of the vascular region represented by the received volumetric image data ( 110 ).
1 . A computer-implemented method of providing a neural network for predicting a three-dimensional shape of an interventional device disposed within a vascular region, the method comprising:
receiving volumetric image data representing a three-dimensional shape of the vascular region;
receiving X-ray image data representing one or more two-dimensional projections of the interventional device within the vascular region;
receiving ground truth interventional device shape data representing a three-dimensional shape of the interventional device within the vascular region corresponding to the one or more two-dimensional projections of the interventional device; and
training a neural network to predict, from the received X-ray image data and the received volumetric image data, a three-dimensional shape of the interventional device constrained by the vascular region, by: inputting the received X-ray image data and the received volumetric image data into the neural network, and adjusting parameters of the neural network based on a first loss function representing a difference between a three-dimensional shape of the interventional device predicted by the neural network, and the received ground truth interventional device shape data, and constraining the adjusting such that the three-dimensional shape of the interventional device predicted by the neural network fits within the three-dimensional shape of the vascular region represented by the received volumetric image data.
2 . The computer-implemented method according to claim 1 , wherein the adjusting parameters of the neural network is based further on a second loss function representing a difference between a two-dimensional projection of the three-dimensional shape of the interventional device predicted by the neural network, and the received X-ray image data; the two-dimensional projection of the three-dimensional shape of the interventional device, and the received X-ray image data being projected onto a common surface.
3 . The computer-implemented method according to claim 1 , further comprising computing an estimated uncertainty of the three-dimensional shape of the interventional device predicted by the neural network.
4 . The computer-implemented method according to claim 1 , wherein the volumetric image data comprises one or more of:
computed tomography image data;
contrast-enhanced computed tomography image data;
3D ultrasound image data;
cone beam computed tomography image data;
magnetic resonance image data;
anatomical atlas model data; and
reconstructed volumetric image data generated by reconstructing X-ray image data representing one or more two-dimensional projections of the vascular region.
5 . The computer-implemented method according to claim 1 , comprising: segmenting the received X-ray image data to provide the one or more two-dimensional projections of the interventional device, and wherein the inputting the received X-ray image data into the neural network comprises inputting the segmented received X-ray image data into the neural network.
6 . The computer-implemented method according to claim 1 , wherein the ground truth interventional device shape data comprises one or more of:
computed tomography image data;
contrast-enhanced computed tomography image data;
cone beam computed tomography image data;
fiber optical shape sensing position data generated by a plurality of fiber optic shape sensors mechanically coupled to the interventional device;
electromagnetic tracking position data generated by one or more electromagnetic tracking sensors or emitters mechanically coupled to the interventional device;
dielectric mapping position data generated by one or more dielectric sensors mechanically coupled to the interventional device; and
ultrasound tracking position data generated by one or more ultrasound tracking sensors or emitters mechanically coupled to the interventional device.
7 . The computer-implemented method according to claim 1 , wherein the training the neural network comprises constraining the adjusting such that the three-dimensional shape of the interventional device predicted by the neural network satisfies one or more mechanical constraints of the interventional device.
8 . The computer-implemented method according to claim 1 , wherein the neural network comprises one or more of: a convolutional neural network, an encoder-decoder network, a generative adversarial network, a capsule network, a regression network, a reinforcement learning agent, a recurrent neural network, a long short-term memory network, a temporal convolutional network, and a transformer.
9 . The computer-implemented method according to claim 1 , wherein the received volumetric image data further represents a three-dimensional shape of an anatomical feature, wherein the received X-ray image data further represents a two-dimensional projection of the anatomical feature; and wherein the training the neural network further comprises training the neural network to predict, from the received X-ray image data, a position of the anatomical feature relative to the three-dimensional shape of the interventional device, and further comprising constraining the adjusting based on a difference between the predicted position of the anatomical feature relative to the three-dimensional shape of the interventional device, and the position of the anatomical feature relative to the three-dimensional shape of the vascular region in the received volumetric image data.
10 . A computer-implemented method of predicting a three-dimensional shape of an interventional device disposed within a vascular region, the method comprising:
receiving volumetric image data representing a three-dimensional shape of the vascular region;
receiving X-ray image data representing one or more two-dimensional projections of the interventional device within the vascular region; and
predicting, from the received X-ray image data and the received volumetric image data, a three-dimensional shape of the interventional device constrained by the vascular region;
wherein a neural network is trained to predict, from one and only one two-dimensional projection of the interventional device within the vascular region, and the received volumetric image data, the three-dimensional shape of the interventional device constrained by the vascular region.
11 . The computer-implemented method according to claim 10 , further comprising projecting the predicted three-dimensional shape of the interventional device, onto at least one surface, to provide a at least one predicted two-dimensional projection of the interventional device.
12 . The computer-implemented method according to claim 11 , wherein the projecting comprises projecting the predicted three-dimensional shape of the interventional device onto a plurality of intersecting surfaces.
13 . A system for predicting a three-dimensional shape of an interventional device disposed within a vascular region; the system comprising one or more processors configured to perform the method according to claim 10 .
14 . A non-transitory computer-readable storage medium having stored a computer program comprising instructions which, when executed by a processor, cause the processor to:
receive volumetric image data representing a three-dimensional shape of the vascular region; receive X-ray image data representing one or more two-dimensional projections of the interventional device within the vascular region; and
predict, from the received X-ray image data and the received volumetric image data, a three-dimensional shape of the interventional device constrained by the vascular region;
wherein a neural network is trained to predict, from one and only one two-dimensional projection of the interventional device within the vascular region, and the received volumetric image data, the three-dimensional shape of the interventional device constrained by the vascular region.