Methods and systems for adaptive radiotherapy treatment planning using deep learning engines
Example methods for adaptive radiotherapy treatment planning using deep learning engines are provided. One example method may comprise obtaining treatment image data associated with a first imaging modality and planning image data associated with a second imaging modality. The treatment image data may be acquired during a treatment phase of a patient. Also, planning image data associated with a second imaging modality may be acquired prior to the treatment phase to generate a treatment plan for the patient. The method may also comprise: in response to determination that an update of the treatment plan is required, processing, using the deep learning engine, the treatment image data and the planning image data to generate output data for updating the treatment plan.
1. A method for a computer system to perform adaptive radiotherapy treatment planning, wherein the method comprises:
obtaining treatment image data associated with a first imaging modality, wherein the treatment image data is acquired during a treatment phase of a patient;
obtaining planning image data associated with a second imaging modality, wherein the planning image data is acquired prior to the treatment phase to generate a treatment plan for the patient;
based on a determination that a difference between the treatment image data and the planning image data does not exceed a predetermined significance threshold;
generating transformed image data associated with the first imaging modality based on (a) the treatment image data associated with the first imaging modality and (b) the planning image data associated with the second imaging modality; and
processing, using a deep learning engine, input image data that includes both the transformed image data and the planning image data to generate output data for updating the treatment plan.
2. The method of claim 1 , wherein generating the transformed treatment image data comprises:
performing image registration to generate the transformed image data by registering the treatment image data against the planning image data.
3. The method of claim 1 , wherein obtaining the treatment image data and the planning image data comprises:
obtaining the planning image data whose difference from the treatment image data does not exceed the predetermined significance threshold relating to at least one of the following: shape, size or position change of a target requiring dose delivery; and shape, size or position change of healthy tissue proximal to the target.
4. The method of claim 1 , wherein processing the transformed image data and the planning image data comprises:
processing, using the deep learning engine with multiple processing pathways associated with respective multiple resolution levels, the input image data that includes the transformed image data and the planning image data to the generate output data.
5. The method of claim 4 , wherein processing the transformed image data and the planning image data comprises:
processing, using a first processing pathway of the multiple processing pathways, the input image data that includes both the transformed image data and the planning image data to generate first feature data associated with a first resolution level;
processing, using a second processing pathway of the multiple processing pathways, input image data that includes both the transformed image data and the planning image data to generate second feature data associated with a second resolution level; and
processing, using a third processing pathway of the multiple processing pathways, input image data that includes both the transformed image data and the planning image data to generate third feature data associated with a third resolution level.
6. The method of claim 5 , wherein processing the transformed image data and the planning image data comprises:
generating (a) a first combined set based on the second feature data and the third feature data, and (b) a second combined set based on the first feature data and the first combined set; and
generating the output data based on the second combined set.
7. The method of claim 1 , wherein obtaining the treatment image data and the planning image data comprises one of the following:
obtaining the treatment image data in the form of cone beam computed tomography (CBCT) image data, and the planning image data in the form of computed tomography (CT) image data, ultrasound image data, magnetic resonance imaging (MRI) image data, positron emission tomography (PET) image data, single photon emission computed tomography (SPECT) or camera image data;
obtaining the treatment image data in the form of CT image data, and the planning image data in the form of CT image data associated with a different energy level, ultrasound image data, MRI image data, PET image data, SPECT image data or camera image data;
obtaining the treatment image data in the form of MRI image data, and the planning image data in the form of CT image data, CBCT image data, ultrasound image data, MRI image data, PET image data, SPECT image data or camera image data;
obtaining the treatment image data in the form of ultrasound image data, and the planning image data in the form of CT image data, CBCT image data, PET image data, MRI image data, SPECT image data or camera image data; and
obtaining the treatment image data in the form of PET image data, and the planning image data in the form of CT image data, CBCT image data, ultrasound image data, MRI image data, SPECT image data or camera image data.
8. The method of claim 1 , wherein the method further comprises:
prior to processing the input image data, training the deep learning engine using two sets of training image data that are acquired using the respective first imaging modality and second imaging modality.
9. The method of claim 1 , wherein the method further comprises:
prior to processing the input image data, training the deep learning engine to perform one of the following using training data associated with past patients: automatic segmentation to generate the output data in the form of structure data associated with the patient, dose prediction to generate the output data in the form of dose data associated with the patient, and treatment delivery data estimation to generate the output data in the form of treatment delivery data for a treatment delivery system.
10. A computer system configured to perform adaptive radiotherapy treatment planning, the computer system comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that, in response to execution by the processor, cause the processor to:
obtain treatment image data associated with a first imaging modality, wherein the treatment image data is acquired during a treatment phase of a patient;
obtain planning image data associated with a second imaging modality, wherein the planning image data is acquired prior to the treatment phase to generate a treatment plan for the patient;
based on a determination that a difference between the treatment image data and the planning image data does not exceed a predetermined significance threshold;
generate transformed image data associated with the first imaging modality based on (a) the treatment image data associated with the first imaging modality and (b) the planning image data associated with the second imaging modality; and
process, using a deep learning engine, input image data that includes both the transformed image data and the planning image data to generate output data for updating the treatment plan.
11. The computer system of claim 10 , wherein the instructions for generating the transformed treatment image data cause the processor to:
perform image registration to generate the transformed image data by registering the treatment image data against the planning image data.
12. The computer system of claim 10 , wherein the instructions for obtaining the treatment image data and the planning image data cause the processor to:
obtain the planning image data whose difference from the treatment image data does not exceed the predetermined significance threshold relating to at least one of the following: shape, size or position change of a target requiring dose delivery; and shape, size or position change of healthy tissue proximal to the target.
13. The computer system of claim 10 , wherein the instructions for processing the transformed image data and the planning image data cause the processor to:
process, using the deep learning engine with multiple processing pathways associated with respective multiple resolution levels, the input image data that includes the transformed image data and the planning image data to the generate output data.
14. The computer system of claim 13 , wherein the instructions for processing the transformed image data and the planning image data cause the processor to:
process, using a first processing pathway of the multiple processing pathways, the input image data that includes both the transformed image data and the planning image data to generate first feature data associated with a first resolution level;
process, using a second processing pathway of the multiple processing pathways, input image data that includes both the transformed image data and the planning image data to generate second feature data associated with a second resolution level; and
process, using a third processing pathway of the multiple processing pathways, input image data that includes both the transformed image data and the planning image data to generate third feature data associated with a third resolution level.
15. The computer system of claim 14 , wherein the instructions for processing the transformed image data and the planning image data cause the processor to:
generate (a) a first combined set based on the second feature data and the third feature data, and (b) a second combined set based on the first feature data and the first combined set; and
generate the output data based on the second combined set.
16. The computer system of claim 10 , wherein the instructions for obtaining the treatment image data and the planning image data, in response to execution by the processor, cause the processor to perform one of the following:
obtain the treatment image data in the form of cone beam computed tomography (CBCT) image data, and the planning image data in the form of computed tomography (CT) image data, ultrasound image data, magnetic resonance imaging (MRI) image data, positron emission tomography (PET) image data, single photon emission computed tomography (SPECT) or camera image data;
obtain the treatment image data in the form of CT image data, and the planning image data in the form of CT image data associated with a different energy level, ultrasound image data, MRI image data, PET image data, SPECT image data or camera image data;
obtain the treatment image data in the form of MRI image data, and the planning image data in the form of CT image data, CBCT image data, ultrasound image data, MRI image data, PET image data, SPECT image data or camera image data;
obtain the treatment image data in the form of ultrasound image data, and the planning image data in the form of CT image data, CBCT image data, PET image data, MRI image data, SPECT image data or camera image data; and
obtain the treatment image data in the form of PET image data, and the planning image data in the form of CT image data, CBCT image data, ultrasound image data, MRI image data, SPECT image data or camera image data.
17. The computer system of claim 10 , wherein the non-transitory computer-readable medium having stored thereon additional instructions that, in response to execution by the processor, cause the processor to:
prior to processing the input image data, train the deep learning engine using two sets of training image data that are acquired using the respective first imaging modality and second imaging modality.
18. The computer system of claim 10 , wherein the non-transitory computer-readable medium having stored thereon additional instructions that, in response to execution by the processor, cause the processor to:
prior to processing the input image data, train the deep learning engine to perform one of the following using training data associated with past patients: automatic segmentation to generate the output data in the form of structure data associated with the patient, dose prediction to generate the output data in the form of dose data associated with the patient, and treatment delivery data estimation to generate the output data in the form of treatment delivery data for a treatment delivery system.