Pixelwise positional embeddings for medical images in vision transformers
Systems and methods for performing a medical imaging analysis task based on pixelwise positionally encoded features are provided. One or more input medical images are received. One or more pixelwise positional embedding images are generated for the one or more input medical images using a spatially varying function. Patches are extracted from the one or more input medical images and the one or more pixelwise positional embedding images. The patches extracted from the one or more input medical images are encoded with corresponding ones of the patches extracted from the one or more pixelwise positional embedding images into pixelwise positionally encoded features. A medical imaging analysis task is performed using a machine learning based network based on the pixelwise positionally encoded features. Results of the medical imaging analysis task are output.
1 . A computer-implemented method comprising:
receiving one or more input medical images;
generating one or more pixelwise positional embedding images for the one or more input medical images using a spatially varying function;
extracting patches from the one or more input medical images and the one or more pixelwise positional embedding images;
encoding the patches extracted from the one or more input medical images with corresponding ones of the patches extracted from the one or more pixelwise positional embedding images into pixelwise positionally encoded features by:
combining each patch extracted from the one or more input medical images with its corresponding patch extracted from the one or more pixelwise positional embedding images, and
separately encoding the combined patches;
performing a medical imaging analysis task using a machine learning based network based on the pixelwise positionally encoded features; and
outputting results of the medical imaging analysis task.
2 . The computer-implemented method of claim 1 , wherein generating one or more pixelwise positional embedding images for the one or more input medical images using a spatially varying function comprises:
sampling a spatially varying function at a location of each pixel of the one or more input medical images.
3 . The computer-implemented method of claim 1 , wherein the spatially varying function is a sinusoidal function.
4 . The computer-implemented method of claim 1 , wherein the spatially varying function is in reference coordinate system defined relative to the one or more input medical images.
5 . The computer-implemented method of claim 4 , wherein the reference coordinate system comprises a physical coordinate system of an image acquisition device that acquired the one or more input medical images.
6 . The computer-implemented method of claim 1 , wherein the one or more input medical images comprises a plurality of input medical images and the one or more pixelwise positional embedding images comprises a plurality of pixelwise positional embedding images.
7 . The computer-implemented method of claim 1 , wherein the one or more input medical images comprises a plurality of input medical images and the one or more pixelwise positional embedding images comprises a single pixelwise positional embedding image.
8 . The computer-implemented method of claim 1 , wherein:
encoding the patches extracted from the one or more input medical images with corresponding ones of the patches extracted from the one or more pixelwise positional embedding images into pixelwise positionally encoded features comprises encoding the patches extracted from the one or more input medical images with patch-wise positionally embedded features to generate patch-wise and pixelwise positionally encoded features; and
performing a medical imaging analysis task using a machine learning based network based on the pixelwise positionally encoded features comprises performing the medical imaging analysis task based on the patch-wise and pixelwise positionally encoded features.
9 . The computer-implemented method of claim 1 , wherein the machine learning based network is a vision transformer network.
10 . An apparatus comprising:
means for receiving one or more input medical images;
means for generating one or more pixelwise positional embedding images for the one or more input medical images using a spatially varying function;
means for extracting patches from the one or more input medical images and the one or more pixelwise positional embedding images;
means for encoding the patches extracted from the one or more input medical images with corresponding ones of the patches extracted from the one or more pixelwise positional embedding images into pixelwise positionally encoded features by:
combining each patch extracted from the one or more input medical images with its corresponding patch extracted from the one or more pixelwise positional embedding images, and
separately encoding the combined patches;
means for performing a medical imaging analysis task using a machine learning based network based on the pixelwise positionally encoded features; and
means for outputting results of the medical imaging analysis task.
11 . The apparatus of claim 10 , wherein the means for generating one or more pixelwise positional embedding images for the one or more input medical images using a spatially varying function comprises:
means for sampling a spatially varying function at a location of each pixel of the one or more input medical images.
12 . The apparatus of claim 10 , wherein the spatially varying function is a sinusoidal function.
13 . The apparatus of claim 10 , wherein the spatially varying function is in reference coordinate system defined relative to the one or more input medical images.
14 . The apparatus of claim 13 , wherein the reference coordinate system comprises a physical coordinate system of an image acquisition device that acquired the one or more input medical images.
15 . A non-transitory computer readable medium storing computer program instructions, the computer program instructions when executed by a processor cause the processor to perform operations comprising:
receiving one or more input medical images;
generating one or more pixelwise positional embedding images for the one or more input medical images using a spatially varying function;
extracting patches from the one or more input medical images and the one or more pixelwise positional embedding images;
encoding the patches extracted from the one or more input medical images with corresponding ones of the patches extracted from the one or more pixelwise positional embedding images into pixelwise positionally encoded features by:
combining each patch extracted from the one or more input medical images with its corresponding patch extracted from the one or more pixelwise positional embedding images, and
separately encoding the combined patches;
performing a medical imaging analysis task using a machine learning based network based on the pixelwise positionally encoded features; and
outputting results of the medical imaging analysis task.
16 . The non-transitory computer readable medium of claim 15 , wherein generating one or more pixelwise positional embedding images for the one or more input medical images using a spatially varying function comprises:
sampling a spatially varying function at a location of each pixel of the one or more input medical images.
17 . The non-transitory computer readable medium of claim 15 , wherein the one or more input medical images comprises a plurality of input medical images and the one or more pixelwise positional embedding images comprises a plurality of pixelwise positional embedding images.
18 . The non-transitory computer readable medium of claim 15 , wherein the one or more input medical images comprises a plurality of input medical images and the one or more pixelwise positional embedding images comprises a single pixelwise positional embedding image.
19 . The non-transitory computer readable medium of claim 15 , wherein:
encoding the patches extracted from the one or more input medical images with corresponding ones of the patches extracted from the one or more pixelwise positional embedding images into pixelwise positionally encoded features comprises encoding the patches extracted from the one or more input medical images with patch-wise positionally embedded features to generate patch-wise and pixelwise positionally encoded features; and
performing a medical imaging analysis task using a machine learning based network based on the pixelwise positionally encoded features comprises performing the medical imaging analysis task based on the patch-wise and pixelwise positionally encoded features.
20 . The non-transitory computer readable medium of claim 15 , wherein the machine learning based network is a vision transformer network.