Methods and systems for radiotherapy treatment planning using deep learning engines
Example methods for radiotherapy treatment planning using deep learning engines are provided. One example method may comprise obtaining first image data associated with a patient; generating first feature data by processing the first image data associated with a first resolution level using a first processing pathway; generating second feature data by processing second image data associated with a second resolution level using a second processing pathway; and generating third feature data by processing third image data associated with a third resolution level using a third processing pathway. The example method may also comprise generating a first combined set of feature data associated with the second resolution level, and a second combined set of feature data associated with the first resolution level based on the first feature data and the first combined set. Further, the example method may comprise generating output data associated with radiotherapy treatment of the patient.
1. A method for a computer system to perform radiotherapy treatment planning using a deep learning engine that includes at least a first processing pathway, a second processing pathway and a third processing pathway, wherein the method comprises:
obtaining first image data associated with a first resolution level and a patient;
generating first feature data by processing the first image data using a first set of one or more convolution layers included in the first processing pathway, wherein the first image data is a first input of the first processing pathway;
generating second feature data by processing second image data associated with a second resolution level using a second set of one or more convolution layers included in the second processing pathway, wherein the second image data is a second input of the second processing pathway and is generated based on the first image data but without being processed by the first set of one or more convolution layers included in the first processing pathway;
downsampling the first image data or the second image data to generate third image data associated with a third resolution level;
generating third feature data by processing the third image data using a third set of one or more convolution layers included in the third processing pathway, wherein the first resolution level, the second resolution level, and the third resolution level are different, and the third image data is a third input of the third processing pathway but without being processed by the first set of one or more convolution layers included in the first processing pathway or the second set of one or more convolution layers included in the second processing pathway;
generating a first combined set of feature data associated with the second resolution level based on the second feature data and the third feature data, and a second combined set of feature data associated with the first resolution level based on the first feature data and the first combined set; and
based on the second combined set, generating output data associated with radiotherapy treatment of the patient.
2. The method of claim 1 , wherein the method further comprises:
upsampling the third feature data from the third resolution level to the second resolution level prior to generating the first combined set.
3. The method of claim 2 , wherein generating the first combined set comprises:
generating the first combined set by processing the upsampled third feature data and the second feature data using a fourth set of one or more convolution layers of the deep learning engine.
4. The method of claim 1 , wherein the method further comprises:
downsampling the first image data to generate the second image data associated with the second resolution level; and
upsampling the first combined set from the second resolution level to the first resolution level prior to generating the second combined set.
5. The method of claim 4 , wherein generating the second combined set comprises:
generating the second combined set by processing the upsampled first combined set and the first feature data using a fifth set of one or more convolution layers of the deep learning engine.
6. The method of claim 1 , wherein generating the output data comprises:
generating the output data by processing the second combined set using one or more mixing layers of the deep learning engine.
7. The method of claim 1 , wherein the method further comprises:
training the deep learning engine to perform one of the following using training data associated with past patients: automatic segmentation to generate the output data in a form of structure data associated with the patient, dose prediction to generate the output data in a form of dose data associated with the patient, and treatment delivery data estimation to generate the output data in a form of treatment delivery data for a treatment delivery system.
8. A non-transitory computer-readable storage medium that includes a set of instructions which, in response to execution by a processor of a computer system, cause the processor to perform a method of perform radiotherapy treatment planning using a deep learning engine that includes a first processing pathway, a second processing pathway and a third processing pathway, wherein the method comprises:
obtaining first image data associated with a first resolution level and a patient;
generating first feature data by processing the first image data using a first set of one or more convolution layers included in the first processing pathway, wherein the first image data is a first input of the first processing pathway;
generating second feature data by processing second image data associated with a second resolution level using a second set of one or more convolution layers included in the second processing pathway, wherein the second image data is a second input of the second processing pathway and is generated based on the first image data but without being processed by the first set of one or more convolution layers included in the first processing pathway;
downsampling the first image data or the second image data to generate third image data associated with a third resolution level;
generating third feature data by processing the third image data using a third set of one or more convolution layers included in the third processing pathway, wherein the first resolution level, the second resolution level and the third resolution level are different, and the third image data is a third input of the third processing pathway but without being processed by the first set of one or more convolution layers included in the first processing pathway or the second set of one or more convolution layers included in the second processing pathway;
generating a first combined set of feature data associated with the second resolution level based on the second feature data and the third feature data, and a second combined set of feature data associated with the first resolution level based on the first feature data and the first combined set; and
based on the second combined set, generating output data associated with radiotherapy treatment of the patient.
9. The non-transitory computer-readable storage medium of claim 8 , wherein the method further comprises:
upsampling the third feature data from the third resolution level to the second resolution level prior to generating the first combined set.
10. The non-transitory computer-readable storage medium of claim 9 , wherein generating the first combined set comprises:
generating the first combined set by processing the upsampled third feature data and the second feature data using a fourth set of one or more convolution layers of the deep learning engine.
11. The non-transitory computer-readable storage medium of claim 8 , wherein the method further comprises:
downsampling the first image data to generate the second image data associated with the second resolution level; and
upsampling the first combined set from the second resolution level to the first resolution level prior to generating the second combined set.
12. The non-transitory computer-readable storage medium of claim 11 , wherein generating the second combined set comprises:
generating the second combined set by processing the upsampled first combined set and the first feature data using a fifth set of one or more convolution layers of the deep learning engine.
13. The non-transitory computer-readable storage medium of claim 8 , wherein generating the output data comprises:
generating the output data by processing the second combined set using one or more mixing layers of the deep learning engine.
14. The non-transitory computer-readable storage medium of claim 8 , wherein the method further comprises:
training the deep learning engine to perform one of the following using training data associated with past patients: automatic segmentation to generate the output data in a form of structure data associated with the patient, dose prediction to generate the output data in a form of dose data associated with the patient, and treatment delivery data estimation to generate the output data in a form of treatment delivery data for a treatment delivery system.
15. A computer system configured to perform radiotherapy treatment planning using a deep learning engine that includes a first processing pathway, a second processing pathway and a third processing pathway, wherein the computer system comprises: a processor and a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to:
obtain first image data associated with a first resolution level and a patient;
generate first feature data by processing the first image data using a first set of one or more convolution layers included in the first processing pathway, wherein the first image data is a first input of the first processing pathway;
generate second feature data by processing the second image data associated with a second resolution level using a second set of one or more convolution layers included in the second processing pathway, wherein the second image data is a second input of the second processing pathway and is generated based on the first image data but without being processed by the first set of one or more convolution layers included in the first processing pathway;
downsample the first image data or the second image data to generate third image data associated with a third resolution level;
generate third feature data by processing the third image data using a third set of one or more convolution layers included in the third processing pathway, wherein the first resolution level, the second resolution level and the third resolution level are different, and the third image data is a third input of the third processing pathway but without being processed by the first set of one or more convolution layers included in the first processing pathway or the second set of one or more convolution layers included in the second processing pathway;
generate a first combined set of feature data associated with the second resolution level based on the second feature data and the third feature data, and a second combined set of feature data associated with the first resolution level based on the first feature data and the first combined set; and
based on the second combined set, generate output data associated with radiotherapy treatment of the patient.
16. The computer system of claim 15 , wherein the instructions further cause the processor to:
upsample the third feature data from the third resolution level to the second resolution level prior to generating the first combined set.
17. The computer system of claim 16 , wherein the instructions for generating the first combined set cause the processor to:
generate the first combined set by processing the upsampled third feature data and the second feature data using a fourth set of one or more convolution layers of the deep learning engine.
18. The computer system of claim 15 , wherein the instructions further cause the processor to:
downsample the first image data to generate the second image data associated with the second resolution level; and
upsample the first combined set from the second resolution level to the first resolution level prior to generating the second combined set.
19. The computer system of claim 18 , wherein the instructions for generating the second combined set cause the processor to:
generate the second combined set by processing the upsampled first combined set and the first feature data using a fifth set of one or more convolution layers of the deep learning engine.
20. The computer system of claim 15 , wherein the instructions for generating the output data cause the processor to:
generate the output data by processing the second combined set using one or more mixing layers of the deep learning engine.
21. The computer system of claim 15 , wherein the instructions further cause the processor to:
train the deep learning engine to perform one of the following using training data associated with past patients: automatic segmentation to generate the output data in a form of structure data associated with the patient, dose prediction to generate the output data in a form of dose data associated with the patient, and treatment delivery data estimation to generate the output data in a form of treatment delivery data for a treatment delivery system.