Generating synthetic x-ray images and object annotations from CT scans for augmenting x-ray abnormality assessment systems

1. A computer implemented method comprising:

receiving an input medical image in a first modality;

generating a synthetic image in a second modality from the input medical image;

upsampling the synthetic image to increase a resolution of the synthetic image;

generating an output image to simulate image processing of the upsampled synthetic image; and

outputting the output image,

wherein:

the input medical image comprises annotations,

generating a synthetic image in a second modality from the input medical image comprises translating the annotations to correspond to the synthetic image, and

upsampling the synthetic image to increase a resolution of the synthetic image comprises resampling the translated annotations to the resolution of the upsampled synthetic image.

2. The computer implemented method of claim 1 , wherein upsampling the synthetic image to increase a resolution of the synthetic image comprises:

performing a first upsampling of the synthetic image to increase a one-dimensional resolution of the synthetic image in a head-feet dimension using a first trained machine learning based model; and

performing a second upsampling of the synthetic image to increase a two-dimensional resolution of the synthetic image using a second trained machine learning based model.

3. The computer implemented method of claim 1 , wherein generating an output image to simulate image processing of the upsampled synthetic image comprises:

generating the output image using a trained machine learning based model.

4. The computer implemented method of claim 3 , wherein the trained machine learning based model is trained using unpaired training images.

5. The computer implemented method of claim 1 , wherein generating a synthetic image in a second modality from the input medical image comprises:

segmenting an anatomical object of interest from the input medical image;

masking regions of the input medical image outside of the segmented anatomical object of interest; and

generating the synthetic image from the masked input medical image.

6. The computer implemented method of claim 1 , wherein the first modality is CT (computed tomography) and the second modality is x-ray, and wherein generating a synthetic image in a second modality from the input medical image comprises:

generating a DRR (digitally reconstructed radiograph) from the input medical image.

7. The computer implemented method of claim 1 , further comprising:

training a machine learning based model for performing a medical imaging analysis task based on the output image.

8. A computer implemented method comprising:

receiving an input medical image in a first modality;

generating a synthetic image in a second modality from the input medical image;

upsampling the synthetic image to increase a resolution of the synthetic image;

generating an output image to simulate image processing of the upsampled synthetic image;

outputting the output image;

generating one or more additional synthetic images in the second modality from the input medical image; and

repeating the upsampling, the generating the output image, and the outputting steps for each respective additional synthetic image of the one or more additional synthetic images using the respective additional synthetic image as the synthetic image to generate a plurality of output images.

9. The computer implemented method of claim 8 , wherein upsampling the synthetic image to increase a resolution of the synthetic image comprises:

performing a first upsampling of the synthetic image to increase a one-dimensional resolution of the synthetic image in a head-feet dimension using a first trained machine learning based model; and

performing a second upsampling of the synthetic image to increase a two-dimensional resolution of the synthetic image using a second trained machine learning based model.

10. An apparatus comprising:

means for receiving an input medical image in a first modality;

means for generating a synthetic image in a second modality from the input medical image;

means for upsampling the synthetic image to increase a resolution of the synthetic image;

means for generating an output image to simulate image processing of the upsampled synthetic image; and

means for outputting the output image,

wherein:

the input medical image comprises annotations,

the means for generating a synthetic image in a second modality from the input medical image comprises means for translating the annotations to correspond to the synthetic image, and

the means for upsampling the synthetic image to increase a resolution of the synthetic image comprises means for resampling the translated annotations to the resolution of the upsampled synthetic image.

11. The apparatus of claim 10 , wherein the means for upsampling the synthetic image to increase a resolution of the synthetic image comprises:

means for performing a first upsampling of the synthetic image to increase a one-dimensional resolution of the synthetic image in a head-feet dimension using a first trained machine learning based model; and

means for performing a second upsampling of the synthetic image to increase a two-dimensional resolution of the synthetic image using a second trained machine learning based model.

12. The apparatus of claim 10 , wherein the means for generating an output image to simulate image processing of the upsampled synthetic image comprises:

means for generating the output image using a trained machine learning based model.

13. The apparatus of claim 12 , wherein the trained machine learning based model is trained using unpaired training images.

14. The apparatus of claim 10 , wherein the means for generating a synthetic image in a second modality from the input medical image comprises:

means for segmenting an anatomical object of interest from the input medical image;

means for masking regions of the input medical image outside of the segmented anatomical object of interest; and

means for generating the synthetic image from the masked input medical image.

15. A non-transitory computer readable medium storing computer program instructions, the computer program instructions when executed by a processor cause the processor to perform operations comprising:

receiving an input medical image in a first modality;

generating a synthetic image in a second modality from the input medical image;

upsampling the synthetic image to increase a resolution of the synthetic image;

generating an output image to simulate image processing of the upsampled synthetic image; and

outputting the output image,

wherein:

the input medical image comprises annotations,

generating a synthetic image in a second modality from the input medical image comprises translating the annotations to correspond to the synthetic image, and

upsampling the synthetic image to increase a resolution of the synthetic image comprises resampling the translated annotations to the resolution of the upsampled synthetic image.

16. The non-transitory computer readable medium of claim 15 , wherein upsampling the synthetic image to increase a resolution of the synthetic image comprises:

performing a first upsampling of the synthetic image to increase a one-dimensional resolution of the synthetic image in a head-feet dimension using a first trained machine learning based model; and

performing a second upsampling of the synthetic image to increase a two-dimensional resolution of the synthetic image using a second trained machine learning based model.

17. The non-transitory computer readable medium of claim 15 , wherein the first modality is CT (computed tomography) and the second modality is x-ray, and wherein generating a synthetic image in a second modality from the input medical image comprises:

generating a DRR (digitally reconstructed radiograph) from the input medical image.

18. The non-transitory computer readable medium of claim 15 , the operations further comprising:

training a machine learning based model for performing a medical imaging analysis task based on the output image.

19. A non-transitory computer readable medium storing computer program instructions, the computer program instructions when executed by a processor cause the processor to perform operations comprising:

receiving an input medical image in a first modality;

generating a synthetic image in a second modality from the input medical image;

upsampling the synthetic image to increase a resolution of the synthetic image;

generating an output image to simulate image processing of the upsampled synthetic image;

outputting the output image;

generating one or more additional synthetic images in the second modality from the input medical image; and

repeating the upsampling, the generating the output image, and the outputting operations for each respective additional synthetic image of the one or more additional synthetic images using the respective additional synthetic image as the synthetic image to generate a plurality of output images.

20. An apparatus comprising:

means for receiving an input medical image in a first modality;

means for generating a synthetic image in a second modality from the input medical image;

means for upsampling the synthetic image to increase a resolution of the synthetic image;

means for generating an output image to simulate image processing of the upsampled synthetic image;

means for outputting the output image;

means for generating one or more additional synthetic images in the second modality from the input medical image; and

means for repeating the means for upsampling, the means for generating the output image, and the means for outputting for each respective additional synthetic image of the one or more additional synthetic images using the respective additional synthetic image as the synthetic image to generate a plurality of output images.