Automatic contour adaptation using three sub-networks of a neural network

1. A method comprising:

receiving first and second images depicting an anatomy of a subject, the first image corresponding to a previous radiotherapy treatment fraction, the second image corresponding to a current radiotherapy treatment fraction;

obtaining a first image segmentation associated with the first image;

applying a trained neural network to adapt the first image segmentation to the anatomy of the subject depicted in the second image, wherein applying the trained neural network comprises:

applying a first sub-network of the trained neural network to the first image and the second image to generate a first initial segmentation estimate representing a deformation of the first image segmentation to fit the anatomy depicted in the second image, the first sub-network being trained to establish a relationship between a plurality of previous treatment fraction images and subsequently obtained treatment fraction images;

applying a second sub-network to the second image to generate a second initial segmentation estimate for the second image, the second sub-network being trained to establish a relationship between a plurality of current treatment fraction images and corresponding ground truth segmentations;

applying a third sub-network to the second image and the first and second initial segmentations to predict a refined segmentation for the second image, the third sub-network being trained to establish a relationship between a plurality of pairs of initial segmentation estimates and the corresponding ground truth segmentations; and

configuring a radiotherapy treatment parameter based on the refined segmentation of the second image.

2. The method of claim 1 , wherein the first, second, and third sub-networks are components of a single deep convolutional neural network for contour adaptation.

3. The method of claim 1 , wherein applying the first sub-network to the first image and the second image comprises:

concatenating the first and second images along a channel dimension to generate concatenated data; and

generating, based on a passing the concatenated data through the first sub-network, a deformation vector field (DVF) that maps pixels or voxels from the first image to the second image.

4. The method of claim 3 , further comprising:

resampling the first image with the segmentation associated with the first image based on the DVF to produce an image-segmentation pair that aligns the first image to the second image.

5. The method of claim 3 , further comprising:

deforming the segmentation associated with the first image based on the DVF.

6. The method of claim 1 , further comprising:

converting one or more contours associated with the first image to label maps or binary segmentations to generate the first image segmentation.

7. The method of claim 1 , wherein the radiotherapy treatment parameter comprises converting the refined segmentation of the second image to one or more contours associated with the second image.

8. The method of claim 1 , further comprising:

determining a first modality of the first image; and

adjusting a second image obtained using a second modality to correspond to the first modality of the first image.

9. The method of claim 1 , wherein configuring the radiotherapy treatment parameter comprises at least one of recalculating dose or adjusting one or more radiotherapy treatment machine parameters.

10. The method of claim 1 , wherein configuring the radiotherapy treatment parameter comprises generating a display of the second image with the refined segmentation of the second image on a graphical user interface.

11. The method of claim 1 , further comprising training the first, second, and third sub-networks simultaneously.

12. The method of claim 1 , further comprising training the first, second, and third sub-networks simultaneously by:

obtaining a pair of training images and corresponding ground truth segmentations representing two treatment fractions;

applying the first sub-network to the pair of images and the ground truth segmentation corresponding to a first training image of the pair of training images to generate a first training initial estimated segmentation;

applying the second sub-network to a second training image of the pair of training images to generate a second training initial estimated segmentation;

applying the third sub-network to the second training image and the first and second training initial estimated segmentations to generate a refined segmentation;

computing a set of cost functions, the set of cost functions including a term which measures a discrepancy between the refined segmentation and the ground truth segmentation; and

adjusting one or more parameters of at least one of the first, second, or third sub-networks based on the computed cost function.

13. The method of claim 12 , wherein computing the set of cost functions further comprises computing an additional cost function that includes a difference between a deformation of a given one of the plurality of previous treatment fraction images and a given one of the subsequently obtained treatment fraction images.

14. The method of claim 12 , wherein computing the set of cost functions further comprises computing an additional cost function that includes a difference between a deformation of a segmentation associated with a given one of the plurality of previous treatment fraction images and a ground truth segmentation of the given one of the subsequently obtained treatment fraction images.

15. The method of claim 12 , wherein computing the set of cost functions further comprises computing an additional cost function that includes a measure of smoothness of a generated deformation field.

16. The method of claim 1 , further comprising training the first, second, and third sub-networks in sequence by first adjusting one or more parameters of the first sub-network to minimize a first set of cost functions, then adjusting one or more parameters of the second sub-network to minimize a second set of cost functions, and then adjusting one or more parameters of the third sub-network to minimize a third set of cost functions.

17. The method of claim 16 , further comprising:

training the first sub-network by:

obtaining a pair of training images and corresponding ground truth segmentations representing two treatment fractions;

applying the first sub-network to the pair of training images and the ground truth segmentation corresponding to a first training image in the pair of training images to generate a first initial estimated segmentation; and

computing a first set of cost functions, the set of cost functions including a term which measures a discrepancy between the first initial estimated segmentation and the ground truth segmentation corresponding to a second training image in the pair of training images; and

training the second sub-network by:

applying the second sub-network to the second training image to generate a second initial estimated segmentation; and

computing a second set of cost functions, the set of cost functions including a term which measures a discrepancy between the second initial estimated segmentation and the ground truth segmentation corresponding to the second training image; and

adjusting one or more parameters of at least one of the first sub-network or the second sub-network based on the computed first or second sets of cost functions.

18. The method of claim 16 , further comprising training the third sub-network by:

applying the third sub-network to the second training image and the first and second training initial estimated segmentations to generate a training refined segmentation; and

computing a third set of cost functions, the set of cost functions including a term which measures a discrepancy between the refined segmentation and the ground truth segmentation corresponding to the second training image.

19. A computing apparatus, the computing apparatus comprising:

a processor; and

a memory storing instructions that, when executed by the processor, configure the apparatus to:

receive first and second images depicting an anatomy of a subject, the first image corresponding to a previous radiotherapy treatment fraction, the second image corresponding to a current radiotherapy treatment fraction;

obtain a first image segmentation associated with the first image;

apply a trained neural network to adapt the first image segmentation to the anatomy of the subject depicted in the second image, wherein applying the trained neural network comprises:

apply a first sub-network of the trained neural network to the first image and the second image to generate a first initial segmentation estimate representing a deformation of the first image segmentation to fit the anatomy depicted in the second image, the first sub-network being trained to establish a relationship between a plurality of previous treatment fraction images and subsequently obtained treatment fraction images;

apply a second sub-network to the second image to generate a second initial segmentation estimate for the second image, the second sub-network being trained to establish a relationship between a plurality of current treatment fraction images and corresponding ground truth segmentations;

apply a third sub-network to the second image and the first and second initial segmentations to predict a refined segmentation for the second image, the third sub-network being trained to establish a relationship between a plurality of pairs of initial segmentation estimates and the corresponding ground truth segmentations; and

configure a radiotherapy treatment parameter based on the refined segmentation of the second image.

20. The computing apparatus of claim 19 , wherein the first, second, and third sub-networks are components of a single deep convolutional neural network for contour adaptation.

21. The computing apparatus of claim 19 , wherein applying the first sub-network to the first image and the second image comprises:

concatenate the first and second images along a channel dimension to generate concatenated data; and

generate, based on a passing the concatenated data through the first sub-network, a deformation vector field (DVF) that maps pixels or voxels from the first image to the second image.

22. The computing apparatus of claim 21 , wherein the instructions further configure the apparatus to:

resample the first image with the segmentation associated with the first image based on the DVF to produce an image-segmentation pair that aligns the first image to the second image.

23. The computing apparatus of claim 21 , wherein the instructions further configure the apparatus to:

deform the segmentation associated with the first image based on the DVF.

24. The computing apparatus of claim 19 , wherein the instructions further configure the apparatus to:

convert one or more contours associated with the first image to label maps or binary segmentations to generate the first image segmentation.

25. The computing apparatus of claim 19 , wherein the radiotherapy treatment parameter comprises convert the refined segmentation of the second image to one or more contours associated with the second image.

26. The computing apparatus of claim 19 , wherein the instructions further configure the apparatus to:

determine a first modality of the first image; and

adjust a second image obtained using a second modality to correspond to the first modality of the first image.

27. The computing apparatus of claim 19 , wherein configuring the radiotherapy treatment parameter comprises at least one of recalculating dose or adjust one or more radiotherapy treatment machine parameters.

28. The computing apparatus of claim 19 , wherein configuring the radiotherapy treatment parameter comprises generating a display of the second image with the refined segmentation of the second image on a graphical user interface.

29. The computing apparatus of claim 19 , wherein the instructions further configure the apparatus to train the first sub-network by adjusting one or more parameters of the first sub-network to minimize a first cost function that includes a difference between a deformation of a given one of the plurality of previous treatment fraction images and a given one of the subsequently obtained treatment fraction images.

30. The computing apparatus of claim 19 , wherein the instructions further configure the apparatus to train the first, second, and third sub-networks simultaneously by:

obtaining a pair of training images and corresponding ground truth segmentations representing two treatment fractions;

applying the first sub-network to the pair of images and the ground truth segmentation corresponding to a first training image of the pair of training images to generate a first training initial estimated segmentation;

applying the second sub-network to a second training image of the pair of training images to generate a second training initial estimated segmentation;

applying the third sub-network to the second training image and the first and second training initial estimated segmentations to generate a training refined segmentation;

computing a set of cost functions, the set of cost functions including a term which measures a discrepancy between the refined segmentation and the ground truth segmentation corresponding to the second training image; and

adjusting one or more parameters of at least one of the first, second, or third sub-networks based on the computed cost function.

31. The computing apparatus of claim 30 , wherein computing the set of cost functions further comprises computing an additional cost function that includes a difference between a deformation of a given one of the plurality of previous treatment fraction images and a given one of the subsequently obtained treatment fraction images.

32. The computing apparatus of claim 30 , wherein computing the set of cost functions further comprises computing an additional cost function that includes a difference between a deformation of a segmentation associated with a given one of the plurality of previous treatment fraction images and a ground truth segmentation of the given one of the subsequently obtained treatment fraction images.

33. The computing apparatus of claim 30 , wherein computing the set of cost functions further comprises computing an additional cost function that includes a measure of smoothness of a generated deformation field.

34. The computing apparatus of claim 19 , wherein the instructions further configure the apparatus to train the first, second, and third sub-networks in sequence by first adjusting one or more parameters of the first sub-network to minimize a first set of cost functions, then adjusting one or more parameters of the second sub-network to minimize a second set of cost functions, and then adjusting one or more parameters of the third sub-network to minimize a third set of cost functions.

35. The computing apparatus of claim 34 , wherein the instructions further configure the apparatus to:

train the first sub-network by:

obtaining a pair of training images and corresponding ground truth segmentations representing two treatment fractions;

applying the first sub-network to the pair of training images and the ground truth segmentation corresponding to a first training image in the pair of training images to generate a first initial estimated segmentation; and

computing a first set of cost functions, the set of cost functions including a term which measures a discrepancy between the first initial estimated segmentation and the ground truth segmentation corresponding to a second training image in the pair of training images; and

train the second sub-network by:

applying the second sub-network to the second training image to generate a second initial estimated segmentation; and

compute a second set of cost functions, the set of cost functions including a term which measures a discrepancy between the second initial estimated segmentation and the ground truth segmentation corresponding to the second training image; and

adjust one or more parameters of at least one of the first sub-network or the second sub-network based on the computed first or second sets of cost functions.

36. The computing apparatus of claim 34 , wherein the instructions further configure the apparatus to train the third sub-network by:

applying the third sub-network to the second training image and the first and second training initial estimated segmentations to generate a training refined segmentation; and

computing a third set of cost functions, the set of cost functions including a term which measures a discrepancy between the training refined segmentation and the ground truth segmentation corresponding to the second training image.

37. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:

receive first and second images depicting an anatomy of a subject, the first image corresponding to a previous radiotherapy treatment fraction, the second image corresponding to a current radiotherapy treatment fraction;

obtain a first image segmentation associated with the first image;

apply a trained neural network to adapt the first image segmentation to the anatomy of the subject depicted in the second image, wherein applying the trained neural network comprises:

applying a first sub-network of the trained neural network to the first image and the second image to generate a first initial segmentation estimate representing a deformation of the first image segmentation to fit the anatomy depicted in the second image, the first sub-network being trained to establish a relationship between a plurality of previous treatment fraction images and subsequently obtained treatment fraction images;

applying a second sub-network to the second image to generate a second initial segmentation estimate for the second image, the second sub-network being trained to establish a relationship between a plurality of current treatment fraction images and corresponding ground truth segmentations;

applying a third sub-network to the second image and the first and second initial segmentations to predict a refined segmentation for the second image, the third sub-network being trained to establish a relationship between a plurality of pairs of initial segmentation estimates and the corresponding ground truth segmentations; and

configuring a radiotherapy treatment parameter based on the refined segmentation of the second image.